[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:04:47 dist-test-slave-rn98 krb5kdc[794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:04:47 dist-test-slave-rn98 krb5kdc[794](info): set up 2 sockets
Sep 25 19:04:47 dist-test-slave-rn98 krb5kdc[794](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 19:04:48.163414 787 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:04:48 dist-test-slave-rn98 krb5kdc[794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:04:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:04:48.285943 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:36731
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:38799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:36731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:48.367960 807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:48.368243 807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:48.368285 807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:48.369519 807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:48.369596 807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:48.369618 807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:48.369638 807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:48.369665 807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:48.371240 807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:38799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:36731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:36731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:48.371572 807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:48.371829 807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:48.377245 812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:48.377218 815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:48.377252 807 server_base.cc:1034] running on GCE node
W20260925 19:04:48.377214 813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:48.377748 807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:48.378059 807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:48.379242 807 hybrid_clock.cc:648] HybridClock initialized: now 1790363088379216 us; error 47 us; skew 500 ppm
Sep 25 19:04:48 dist-test-slave-rn98 krb5kdc[794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:48.381868 807 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:04:48.382365 807 webserver.cc:460] Webserver started at http://127.0.196.254:46417/ using document root <none> and password file <none>
I20260925 19:04:48.382663 807 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:48.382748 807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:48.382910 807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:48.384008 807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "9450704376d94d5a9f6f5b2805a2c5f3"
format_stamp: "Formatted at 2026-09-25 19:04:48 on dist-test-slave-rn98"
server_key: "907bbcb7226bc4fbc30e448fabb0d0a3"
server_key_iv: "698d0cb2a96d43b33b17939bf564c613"
server_key_version: "encryptionkey@0"
I20260925 19:04:48.384411 807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "9450704376d94d5a9f6f5b2805a2c5f3"
format_stamp: "Formatted at 2026-09-25 19:04:48 on dist-test-slave-rn98"
server_key: "907bbcb7226bc4fbc30e448fabb0d0a3"
server_key_iv: "698d0cb2a96d43b33b17939bf564c613"
server_key_version: "encryptionkey@0"
I20260925 19:04:48.385910 807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:04:48.386902 822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:48.387120 807 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:04:48.387238 807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "9450704376d94d5a9f6f5b2805a2c5f3"
format_stamp: "Formatted at 2026-09-25 19:04:48 on dist-test-slave-rn98"
server_key: "907bbcb7226bc4fbc30e448fabb0d0a3"
server_key_iv: "698d0cb2a96d43b33b17939bf564c613"
server_key_version: "encryptionkey@0"
I20260925 19:04:48.387336 807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:48.398150 807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:48.400605 807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:48.400797 807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:48.405287 807 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:36731
I20260925 19:04:48.405349 874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:36731 every 8 connection(s)
I20260925 19:04:48.405754 807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:48.406781 875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:48.409122 875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9450704376d94d5a9f6f5b2805a2c5f3: Bootstrap starting.
I20260925 19:04:48.409787 875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9450704376d94d5a9f6f5b2805a2c5f3: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:48.410087 875 log.cc:824] T 00000000000000000000000000000000 P 9450704376d94d5a9f6f5b2805a2c5f3: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:48.410914 875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9450704376d94d5a9f6f5b2805a2c5f3: No bootstrap required, opened a new log
I20260925 19:04:48.410972 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 807
I20260925 19:04:48.411118 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:04:48.411748 787 external_mini_cluster.cc:1428] Setting key ba51969d0841eed1e9246ea5819afa89
I20260925 19:04:48.412981 875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9450704376d94d5a9f6f5b2805a2c5f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9450704376d94d5a9f6f5b2805a2c5f3" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 36731 } }
I20260925 19:04:48.413158 875 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9450704376d94d5a9f6f5b2805a2c5f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:48.413204 875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9450704376d94d5a9f6f5b2805a2c5f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9450704376d94d5a9f6f5b2805a2c5f3, State: Initialized, Role: FOLLOWER
I20260925 19:04:48.413372 875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9450704376d94d5a9f6f5b2805a2c5f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9450704376d94d5a9f6f5b2805a2c5f3" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 36731 } }
I20260925 19:04:48.413470 875 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9450704376d94d5a9f6f5b2805a2c5f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:04:48.413513 875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9450704376d94d5a9f6f5b2805a2c5f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:04:48.413563 875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9450704376d94d5a9f6f5b2805a2c5f3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:48.414388 875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9450704376d94d5a9f6f5b2805a2c5f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9450704376d94d5a9f6f5b2805a2c5f3" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 36731 } }
I20260925 19:04:48.414511 875 leader_election.cc:302] T 00000000000000000000000000000000 P 9450704376d94d5a9f6f5b2805a2c5f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9450704376d94d5a9f6f5b2805a2c5f3; no voters:
I20260925 19:04:48.414705 875 leader_election.cc:288] T 00000000000000000000000000000000 P 9450704376d94d5a9f6f5b2805a2c5f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:04:48.414733 880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9450704376d94d5a9f6f5b2805a2c5f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:48.414840 880 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9450704376d94d5a9f6f5b2805a2c5f3 [term 1 LEADER]: Becoming Leader. State: Replica: 9450704376d94d5a9f6f5b2805a2c5f3, State: Running, Role: LEADER
I20260925 19:04:48.414919 880 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9450704376d94d5a9f6f5b2805a2c5f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9450704376d94d5a9f6f5b2805a2c5f3" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 36731 } }
I20260925 19:04:48.414983 875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9450704376d94d5a9f6f5b2805a2c5f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:48.415225 880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9450704376d94d5a9f6f5b2805a2c5f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9450704376d94d5a9f6f5b2805a2c5f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9450704376d94d5a9f6f5b2805a2c5f3" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 36731 } } }
I20260925 19:04:48.415330 880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9450704376d94d5a9f6f5b2805a2c5f3 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:48.415474 880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9450704376d94d5a9f6f5b2805a2c5f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9450704376d94d5a9f6f5b2805a2c5f3. Latest consensus state: current_term: 1 leader_uuid: "9450704376d94d5a9f6f5b2805a2c5f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9450704376d94d5a9f6f5b2805a2c5f3" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 36731 } } }
I20260925 19:04:48.415616 880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9450704376d94d5a9f6f5b2805a2c5f3 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:48.415949 885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:04:48.416601 885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:48.418633 885 catalog_manager.cc:1347] Generated new cluster ID: 53581c33416644adbca4f5b61ea5331a
I20260925 19:04:48.418684 885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:04:48.422808 885 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:48.423298 885 catalog_manager.cc:1504] Loading token signing keys...
Sep 25 19:04:48 dist-test-slave-rn98 krb5kdc[794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:04:48.428414 885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9450704376d94d5a9f6f5b2805a2c5f3: Generated new TSK 0
I20260925 19:04:48.428584 885 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:04:48.429517 878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:48.412699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45790 (local address 127.0.196.254:36731)
0925 19:04:48.412889 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:48.412900 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:48.413136 (+ 236us) server_negotiation.cc:407] Connection header received
0925 19:04:48.415227 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:48.415233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:48.415290 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:48.416152 (+ 862us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:48.417829 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:48.418951 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:48.420445 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:48.421042 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:48.421142 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:48.425424 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:48.425429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:48.425429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:48.425442 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:48.427489 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:48.428017 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:48.428020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:48.428021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:48.428150 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:48.428458 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:48.428459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:48.428460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:48.428746 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:48.428870 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:48.429185 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 19:04:48.429299 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for kudu/127.0.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
I20260925 19:04:48.465713 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:0
--local_ip_for_outbound_sockets=127.0.196.193
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:36731
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:38799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:48.550869 901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:48.551126 901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:48.551169 901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:48.552395 901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:48.552471 901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:48.552520 901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:04:48.554497 901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:38799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:36731
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:48.554831 901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:48.555081 901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:48.560292 907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:48.560292 909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:48.560359 906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:48.560493 901 server_base.cc:1034] running on GCE node
I20260925 19:04:48.560695 901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:48.560904 901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:48.562088 901 hybrid_clock.cc:648] HybridClock initialized: now 1790363088562070 us; error 36 us; skew 500 ppm
Sep 25 19:04:48 dist-test-slave-rn98 krb5kdc[794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:48.564764 901 init.cc:377] Logged in from keytab as kudu/127.0.196.193@KRBTEST.COM (short username kudu)
I20260925 19:04:48.565234 901 webserver.cc:460] Webserver started at http://127.0.196.193:46709/ using document root <none> and password file <none>
I20260925 19:04:48.565443 901 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:48.565485 901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:48.565577 901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:48.566589 901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-0/data/instance:
uuid: "e6843daf87314390b95c97b7990a8baf"
format_stamp: "Formatted at 2026-09-25 19:04:48 on dist-test-slave-rn98"
server_key: "80530fa654060cf47a20de9f6b1a009d"
server_key_iv: "4e1aa399e1571f8b0056dd57a65b8cf4"
server_key_version: "encryptionkey@0"
I20260925 19:04:48.566995 901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-0/wal/instance:
uuid: "e6843daf87314390b95c97b7990a8baf"
format_stamp: "Formatted at 2026-09-25 19:04:48 on dist-test-slave-rn98"
server_key: "80530fa654060cf47a20de9f6b1a009d"
server_key_iv: "4e1aa399e1571f8b0056dd57a65b8cf4"
server_key_version: "encryptionkey@0"
I20260925 19:04:48.568240 901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:04:48.568928 916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:48.569092 901 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:04:48.569195 901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "e6843daf87314390b95c97b7990a8baf"
format_stamp: "Formatted at 2026-09-25 19:04:48 on dist-test-slave-rn98"
server_key: "80530fa654060cf47a20de9f6b1a009d"
server_key_iv: "4e1aa399e1571f8b0056dd57a65b8cf4"
server_key_version: "encryptionkey@0"
I20260925 19:04:48.569272 901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:48.585440 901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:48.588047 901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:48.588227 901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:48.588490 901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:04:48.588817 901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:48.588873 901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:48.588938 901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:48.588967 901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:48.595733 901 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:35649
I20260925 19:04:48.595783 1029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:35649 every 8 connection(s)
I20260925 19:04:48.596119 901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
I20260925 19:04:48.601135 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 901
I20260925 19:04:48.601230 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-0/wal/instance
I20260925 19:04:48.601429 787 external_mini_cluster.cc:1428] Setting key aa79258c7e2c26de500af4b541302ab7
Sep 25 19:04:48 dist-test-slave-rn98 krb5kdc[794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:04:48.606501 878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:48.596917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:60427 (local address 127.0.196.254:36731)
0925 19:04:48.596972 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:48.596974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:48.597186 (+ 212us) server_negotiation.cc:407] Connection header received
0925 19:04:48.597649 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:48.597650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:48.597659 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:48.597715 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:48.598745 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:48.599268 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:48.600283 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:48.600494 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:48.600555 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:48.603125 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:48.603127 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:48.603127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:48.603135 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:48.604812 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:48.605354 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:48.605355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:48.605356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:48.605409 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:48.605723 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:48.605724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:48.605724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:48.605841 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:48.605895 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:48.606325 (+ 430us) server_negotiation.cc:299] Negotiation successful
0925 19:04:48.606436 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:04:48.606798 1032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:48.596878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:36731 (local address 127.0.196.193:60427)
0925 19:04:48.597053 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:48.597091 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:48.597564 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:48.597771 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:48.597775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:48.598382 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:48.598585 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:48.598590 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:48.599368 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:48.599369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:48.600191 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:48.600193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:48.600572 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:48.600573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:48.600678 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:48.601395 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:48.601404 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:48.603036 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:48.604880 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:48.604887 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:48.604890 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:48.605271 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:48.605479 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:48.605480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:48.605481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:48.605655 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:48.605913 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:48.605920 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:48.606114 (+ 194us) client_negotiation.cc:769] Sending connection context
0925 19:04:48.606267 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 19:04:48.606484 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260925 19:04:48.607406 1030 heartbeater.cc:347] Connected to a master server at 127.0.196.254:36731
I20260925 19:04:48.607558 1030 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:48.607798 1030 heartbeater.cc:511] Master 127.0.196.254:36731 requested a full tablet report, sending...
I20260925 19:04:48.608467 839 ts_manager.cc:194] Registered new tserver with Master: e6843daf87314390b95c97b7990a8baf (127.0.196.193:35649)
I20260925 19:04:48.609143 839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.193@KRBTEST.COM'} at 127.0.196.193:60427
WARNING: no policy specified for kudu/127.0.196.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
I20260925 19:04:48.630434 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:0
--local_ip_for_outbound_sockets=127.0.196.194
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:36731
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:38799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:48.713258 1035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:48.713511 1035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:48.713553 1035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:48.714797 1035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:48.714860 1035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:48.714905 1035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:04:48.716656 1035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:38799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:36731
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:48.716899 1035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:48.717146 1035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:48.722371 1040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:48.722424 1043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:48.722414 1041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:48.722599 1035 server_base.cc:1034] running on GCE node
I20260925 19:04:48.722812 1035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:48.723011 1035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:48.724176 1035 hybrid_clock.cc:648] HybridClock initialized: now 1790363088724170 us; error 36 us; skew 500 ppm
Sep 25 19:04:48 dist-test-slave-rn98 krb5kdc[794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:48.726771 1035 init.cc:377] Logged in from keytab as kudu/127.0.196.194@KRBTEST.COM (short username kudu)
I20260925 19:04:48.727166 1035 webserver.cc:460] Webserver started at http://127.0.196.194:42463/ using document root <none> and password file <none>
I20260925 19:04:48.727344 1035 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:48.727381 1035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:48.727466 1035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:48.728864 1035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-1/data/instance:
uuid: "db64518eba7746e581ff6f0b3798580c"
format_stamp: "Formatted at 2026-09-25 19:04:48 on dist-test-slave-rn98"
server_key: "c03c9b5ca7a3e381c348d7cef65f895c"
server_key_iv: "7df0f0a8cb06ddeafa6f1306f66126b5"
server_key_version: "encryptionkey@0"
I20260925 19:04:48.729189 1035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-1/wal/instance:
uuid: "db64518eba7746e581ff6f0b3798580c"
format_stamp: "Formatted at 2026-09-25 19:04:48 on dist-test-slave-rn98"
server_key: "c03c9b5ca7a3e381c348d7cef65f895c"
server_key_iv: "7df0f0a8cb06ddeafa6f1306f66126b5"
server_key_version: "encryptionkey@0"
I20260925 19:04:48.730434 1035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 19:04:48.731207 1050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:48.731387 1035 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:48.731479 1035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "db64518eba7746e581ff6f0b3798580c"
format_stamp: "Formatted at 2026-09-25 19:04:48 on dist-test-slave-rn98"
server_key: "c03c9b5ca7a3e381c348d7cef65f895c"
server_key_iv: "7df0f0a8cb06ddeafa6f1306f66126b5"
server_key_version: "encryptionkey@0"
I20260925 19:04:48.731541 1035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:48.743875 1035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:48.745965 1035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:48.746083 1035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:48.746309 1035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:04:48.746694 1035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:48.746733 1035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:48.746757 1035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:48.746801 1035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:48.752852 1035 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:41169
I20260925 19:04:48.752929 1163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:41169 every 8 connection(s)
I20260925 19:04:48.753261 1035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
I20260925 19:04:48.755683 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 1035
I20260925 19:04:48.755777 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-1/wal/instance
I20260925 19:04:48.755932 787 external_mini_cluster.cc:1428] Setting key ea16b1768d89c9abe962fde4dc75a376
Sep 25 19:04:48 dist-test-slave-rn98 krb5kdc[794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.196.195@KRBTEST.COM; defaulting to no policy
I20260925 19:04:48.764189 878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:48.754058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:54227 (local address 127.0.196.254:36731)
0925 19:04:48.754126 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:48.754128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:48.754379 (+ 251us) server_negotiation.cc:407] Connection header received
0925 19:04:48.754956 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:48.754957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:48.754967 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:48.755028 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:48.756244 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:48.756945 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:48.758389 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:48.758742 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:48.758865 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:48.761301 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:48.761303 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:48.761303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:48.761311 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:48.762799 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:48.763228 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:48.763230 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:48.763230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:48.763289 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:48.763540 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:48.763541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:48.763542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:48.763659 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:48.763731 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:48.764006 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 19:04:48.764093 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 19:04:48.764391 1166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:48.754031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:36731 (local address 127.0.196.194:54227)
0925 19:04:48.754211 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:48.754253 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:48.754863 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:48.755085 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:48.755089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:48.755818 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:48.756117 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:48.756123 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:48.757037 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:48.757038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:48.758294 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:48.758296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:48.758863 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:48.758865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:48.758971 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:48.759524 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:48.759539 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:48.761192 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:48.762896 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:48.762903 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:48.762905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:48.763141 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:48.763352 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:48.763353 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:48.763354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:48.763472 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:48.763723 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:48.763729 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:48.763875 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 19:04:48.763988 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 19:04:48.764152 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":105,"threads_started":1}
I20260925 19:04:48.764926 1164 heartbeater.cc:347] Connected to a master server at 127.0.196.254:36731
I20260925 19:04:48.765054 1164 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:48.765241 1164 heartbeater.cc:511] Master 127.0.196.254:36731 requested a full tablet report, sending...
I20260925 19:04:48.765652 839 ts_manager.cc:194] Registered new tserver with Master: db64518eba7746e581ff6f0b3798580c (127.0.196.194:41169)
I20260925 19:04:48.766088 839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.194@KRBTEST.COM'} at 127.0.196.194:54227
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
I20260925 19:04:48.781723 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:0
--local_ip_for_outbound_sockets=127.0.196.195
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:36731
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:38799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:48.865509 1169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:48.865729 1169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:48.865754 1169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:48.866961 1169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:48.867007 1169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:48.867034 1169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:04:48.868521 1169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:38799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:36731
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:48.868778 1169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:48.869014 1169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:48.874013 1174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:48.874042 1175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:48.874013 1177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:48.874217 1169 server_base.cc:1034] running on GCE node
I20260925 19:04:48.874413 1169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:48.874708 1169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:48.875873 1169 hybrid_clock.cc:648] HybridClock initialized: now 1790363088875852 us; error 36 us; skew 500 ppm
Sep 25 19:04:48 dist-test-slave-rn98 krb5kdc[794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:48.878407 1169 init.cc:377] Logged in from keytab as kudu/127.0.196.195@KRBTEST.COM (short username kudu)
I20260925 19:04:48.878995 1169 webserver.cc:460] Webserver started at http://127.0.196.195:41675/ using document root <none> and password file <none>
I20260925 19:04:48.879240 1169 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:48.879303 1169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:48.879459 1169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:48.880530 1169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-2/data/instance:
uuid: "029131ffd65d4119bc7ecaba4c88c516"
format_stamp: "Formatted at 2026-09-25 19:04:48 on dist-test-slave-rn98"
server_key: "8b622022aac16752de2dab4489ca8486"
server_key_iv: "0f5c242473e44084dcf659d1a32b227b"
server_key_version: "encryptionkey@0"
I20260925 19:04:48.880918 1169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-2/wal/instance:
uuid: "029131ffd65d4119bc7ecaba4c88c516"
format_stamp: "Formatted at 2026-09-25 19:04:48 on dist-test-slave-rn98"
server_key: "8b622022aac16752de2dab4489ca8486"
server_key_iv: "0f5c242473e44084dcf659d1a32b227b"
server_key_version: "encryptionkey@0"
I20260925 19:04:48.882242 1169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:04:48.883082 1184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:48.883257 1169 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:04:48.883355 1169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "029131ffd65d4119bc7ecaba4c88c516"
format_stamp: "Formatted at 2026-09-25 19:04:48 on dist-test-slave-rn98"
server_key: "8b622022aac16752de2dab4489ca8486"
server_key_iv: "0f5c242473e44084dcf659d1a32b227b"
server_key_version: "encryptionkey@0"
I20260925 19:04:48.883435 1169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:48.892488 1169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:48.895144 1169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:48.895316 1169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:48.895566 1169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:04:48.895901 1169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:48.895955 1169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:48.895988 1169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:48.896018 1169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:48.902311 1169 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:43137
I20260925 19:04:48.902355 1297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:43137 every 8 connection(s)
I20260925 19:04:48.902753 1169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
I20260925 19:04:48.906497 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 1169
I20260925 19:04:48.906574 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790363087086945-787-0/minicluster-data/ts-2/wal/instance
I20260925 19:04:48.906782 787 external_mini_cluster.cc:1428] Setting key a1480a0880eb4d78f407816ea3e0aeac
Sep 25 19:04:48 dist-test-slave-rn98 krb5kdc[794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:04:48.912400 878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:48.903453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:52957 (local address 127.0.196.254:36731)
0925 19:04:48.903501 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:48.903503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:48.903705 (+ 202us) server_negotiation.cc:407] Connection header received
0925 19:04:48.904197 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:48.904198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:48.904207 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:48.904269 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:48.905313 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:48.905831 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:48.906964 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:48.907196 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:48.907297 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:48.909680 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:48.909682 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:48.909682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:48.909689 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:48.911023 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:48.911498 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:48.911499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:48.911500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:48.911578 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:48.911805 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:48.911806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:48.911806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:48.911915 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:48.911967 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:48.912256 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 19:04:48.912320 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 19:04:48.912683 1300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:48.903430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:36731 (local address 127.0.196.195:52957)
0925 19:04:48.903591 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:48.903633 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:48.904105 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:48.904329 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:48.904333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:48.904914 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:48.905110 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:48.905115 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:48.905918 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:48.905919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:48.906882 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:48.906885 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:48.907327 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:48.907328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:48.907479 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:48.908096 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:48.908108 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:48.909596 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:48.911118 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:48.911124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:48.911126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:48.911419 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:48.911648 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:48.911648 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:48.911648 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:48.911750 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:48.911996 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:48.912001 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:48.912138 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 19:04:48.912231 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 19:04:48.912394 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":94,"threads_started":1}
I20260925 19:04:48.913256 1298 heartbeater.cc:347] Connected to a master server at 127.0.196.254:36731
I20260925 19:04:48.913373 1298 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:48.913554 1298 heartbeater.cc:511] Master 127.0.196.254:36731 requested a full tablet report, sending...
I20260925 19:04:48.913949 839 ts_manager.cc:194] Registered new tserver with Master: 029131ffd65d4119bc7ecaba4c88c516 (127.0.196.195:43137)
I20260925 19:04:48.914321 839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.195@KRBTEST.COM'} at 127.0.196.195:52957
I20260925 19:04:48.918835 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 19:04:48 dist-test-slave-rn98 krb5kdc[794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363088, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 19:04:48 dist-test-slave-rn98 krb5kdc[794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363088, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.196.193@KRBTEST.COM
I20260925 19:04:48.943868 1305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:48.935207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52174 (local address 127.0.196.193:35649)
0925 19:04:48.935553 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:48.935558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:48.935571 (+ 13us) server_negotiation.cc:407] Connection header received
0925 19:04:48.935604 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:48.935605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:48.935619 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:48.935685 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:48.936280 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:48.937029 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:48.937747 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:48.938096 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:48.938157 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:48.940765 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:48.940774 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:48.940774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:48.940782 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:48.942694 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:48.943058 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:48.943061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:48.943061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:48.943122 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:48.943314 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:48.943314 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:48.943315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:48.943475 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:48.943612 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:48.943648 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 19:04:48.943714 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":238,"threads_started":1}
W20260925 19:04:48.944281 944 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52174
Sep 25 19:04:48 dist-test-slave-rn98 krb5kdc[794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 19:04:48 dist-test-slave-rn98 krb5kdc[794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.193@KRBTEST.COM
I20260925 19:04:48.967140 1305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:48.959991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52180 (local address 127.0.196.193:35649)
0925 19:04:48.960057 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:48.960060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:48.960211 (+ 151us) server_negotiation.cc:407] Connection header received
0925 19:04:48.960242 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:48.960243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:48.960256 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:48.960321 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:48.960865 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:48.961381 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:48.962092 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:48.962285 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:48.962379 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:48.964603 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:48.964604 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:48.964605 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:48.964611 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:48.966061 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:48.966386 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:48.966387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:48.966387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:48.966448 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:48.966682 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:48.966683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:48.966683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:48.966778 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:48.966889 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:48.966986 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 19:04:48.967047 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 19:04:48.967733 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 901
I20260925 19:04:48.973397 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 1035
I20260925 19:04:48.978299 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 1169
I20260925 19:04:48.982808 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 807
2026-09-25T19:04:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1903 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:04:49 dist-test-slave-rn98 krb5kdc[1312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:04:49 dist-test-slave-rn98 krb5kdc[1312](info): set up 2 sockets
Sep 25 19:04:49 dist-test-slave-rn98 krb5kdc[1312](info): commencing operation
krb5kdc: starting...
W20260925 19:04:50.094245 787 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:04:50 dist-test-slave-rn98 krb5kdc[1312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:04:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:04:50.215343 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:42115
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:41555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:42115
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:50.298054 1325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:50.298290 1325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:50.298316 1325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:50.299528 1325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:50.299578 1325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:50.299593 1325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:50.299606 1325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:50.299618 1325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:50.301291 1325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:41555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:42115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:42115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:50.301512 1325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:50.301707 1325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:50.306850 1330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:50.306859 1331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:50.306872 1333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:50.307253 1325 server_base.cc:1034] running on GCE node
I20260925 19:04:50.307614 1325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:50.307943 1325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:50.309134 1325 hybrid_clock.cc:648] HybridClock initialized: now 1790363090309125 us; error 38 us; skew 500 ppm
Sep 25 19:04:50 dist-test-slave-rn98 krb5kdc[1312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:50.312001 1325 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:04:50.312584 1325 webserver.cc:460] Webserver started at http://127.0.196.254:36927/ using document root <none> and password file <none>
I20260925 19:04:50.312855 1325 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:50.312925 1325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:50.313077 1325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:50.314195 1325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "5164fccabfc94c058dee0f26aad1847b"
format_stamp: "Formatted at 2026-09-25 19:04:50 on dist-test-slave-rn98"
server_key: "8079215ff696de26e5e84b7d83273c85"
server_key_iv: "2f83e9fcdc0fbe2c643c9b87481c0ad9"
server_key_version: "encryptionkey@0"
I20260925 19:04:50.314582 1325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "5164fccabfc94c058dee0f26aad1847b"
format_stamp: "Formatted at 2026-09-25 19:04:50 on dist-test-slave-rn98"
server_key: "8079215ff696de26e5e84b7d83273c85"
server_key_iv: "2f83e9fcdc0fbe2c643c9b87481c0ad9"
server_key_version: "encryptionkey@0"
I20260925 19:04:50.316025 1325 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:04:50.316847 1340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:50.317028 1325 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:50.317137 1325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "5164fccabfc94c058dee0f26aad1847b"
format_stamp: "Formatted at 2026-09-25 19:04:50 on dist-test-slave-rn98"
server_key: "8079215ff696de26e5e84b7d83273c85"
server_key_iv: "2f83e9fcdc0fbe2c643c9b87481c0ad9"
server_key_version: "encryptionkey@0"
I20260925 19:04:50.317214 1325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:50.341920 1325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:50.344099 1325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:50.344285 1325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:50.348419 1325 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:42115
I20260925 19:04:50.348469 1392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:42115 every 8 connection(s)
I20260925 19:04:50.348860 1325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:50.349721 1393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:50.350576 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 1325
I20260925 19:04:50.350713 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:04:50.350879 787 external_mini_cluster.cc:1428] Setting key aa530b75dcbcf40ccfc26157a90d16af
I20260925 19:04:50.353098 1393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5164fccabfc94c058dee0f26aad1847b: Bootstrap starting.
I20260925 19:04:50.353796 1393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5164fccabfc94c058dee0f26aad1847b: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:50.354103 1393 log.cc:824] T 00000000000000000000000000000000 P 5164fccabfc94c058dee0f26aad1847b: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:50.354902 1393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5164fccabfc94c058dee0f26aad1847b: No bootstrap required, opened a new log
I20260925 19:04:50.356956 1393 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5164fccabfc94c058dee0f26aad1847b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5164fccabfc94c058dee0f26aad1847b" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 42115 } }
I20260925 19:04:50.357096 1393 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5164fccabfc94c058dee0f26aad1847b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:50.357118 1393 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5164fccabfc94c058dee0f26aad1847b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5164fccabfc94c058dee0f26aad1847b, State: Initialized, Role: FOLLOWER
I20260925 19:04:50.357265 1393 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5164fccabfc94c058dee0f26aad1847b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5164fccabfc94c058dee0f26aad1847b" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 42115 } }
I20260925 19:04:50.357352 1393 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5164fccabfc94c058dee0f26aad1847b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:04:50.357375 1393 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5164fccabfc94c058dee0f26aad1847b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:04:50.357405 1393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5164fccabfc94c058dee0f26aad1847b [term 0 FOLLOWER]: Advancing to term 1
Sep 25 19:04:50 dist-test-slave-rn98 krb5kdc[1312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:04:50.357978 1393 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5164fccabfc94c058dee0f26aad1847b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5164fccabfc94c058dee0f26aad1847b" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 42115 } }
I20260925 19:04:50.358093 1393 leader_election.cc:302] T 00000000000000000000000000000000 P 5164fccabfc94c058dee0f26aad1847b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5164fccabfc94c058dee0f26aad1847b; no voters:
I20260925 19:04:50.358347 1393 leader_election.cc:288] T 00000000000000000000000000000000 P 5164fccabfc94c058dee0f26aad1847b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:04:50.358376 1398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5164fccabfc94c058dee0f26aad1847b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:50.358487 1398 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5164fccabfc94c058dee0f26aad1847b [term 1 LEADER]: Becoming Leader. State: Replica: 5164fccabfc94c058dee0f26aad1847b, State: Running, Role: LEADER
I20260925 19:04:50.358589 1398 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5164fccabfc94c058dee0f26aad1847b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5164fccabfc94c058dee0f26aad1847b" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 42115 } }
I20260925 19:04:50.358649 1393 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5164fccabfc94c058dee0f26aad1847b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:50.359311 1400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5164fccabfc94c058dee0f26aad1847b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5164fccabfc94c058dee0f26aad1847b. Latest consensus state: current_term: 1 leader_uuid: "5164fccabfc94c058dee0f26aad1847b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5164fccabfc94c058dee0f26aad1847b" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 42115 } } }
I20260925 19:04:50.359397 1400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5164fccabfc94c058dee0f26aad1847b [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:50.359541 1399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5164fccabfc94c058dee0f26aad1847b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5164fccabfc94c058dee0f26aad1847b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5164fccabfc94c058dee0f26aad1847b" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 42115 } } }
I20260925 19:04:50.359614 1399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5164fccabfc94c058dee0f26aad1847b [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:50.359735 1407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:04:50.360258 1407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:50.362007 1407 catalog_manager.cc:1347] Generated new cluster ID: 7a91878f3c934b52b434acc8e90e7684
I20260925 19:04:50.362061 1407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:04:50.362918 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:50.351489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39808 (local address 127.0.196.254:42115)
0925 19:04:50.351665 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:50.351675 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:50.351736 (+ 61us) server_negotiation.cc:407] Connection header received
0925 19:04:50.352269 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:50.352279 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:50.352349 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:50.352510 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:50.353060 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.354436 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.355347 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.355791 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.355920 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:50.358497 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:50.358502 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:50.358503 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:50.358515 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:50.360817 (+ 2302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:50.361274 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:50.361277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:50.361278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:50.361396 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:50.361629 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:50.361630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:50.361631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:50.361957 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:50.362489 (+ 532us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:50.362562 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 19:04:50.362722 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
I20260925 19:04:50.378059 1407 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:50.378935 1407 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:50.382689 1407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5164fccabfc94c058dee0f26aad1847b: Generated new TSK 0
I20260925 19:04:50.382889 1407 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
I20260925 19:04:50.417735 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:0
--local_ip_for_outbound_sockets=127.0.196.193
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:42115
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:41555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:50.499708 1419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:50.499927 1419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:50.499953 1419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:50.501060 1419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:50.501119 1419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:50.501147 1419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:04:50.502605 1419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:41555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:42115
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:50.502904 1419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:50.503163 1419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:50.508332 1424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:50.508442 1419 server_base.cc:1034] running on GCE node
W20260925 19:04:50.508303 1427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:50.508309 1425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:50.508765 1419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:50.509032 1419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:50.510224 1419 hybrid_clock.cc:648] HybridClock initialized: now 1790363090510214 us; error 37 us; skew 500 ppm
Sep 25 19:04:50 dist-test-slave-rn98 krb5kdc[1312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:50.512792 1419 init.cc:377] Logged in from keytab as kudu/127.0.196.193@KRBTEST.COM (short username kudu)
I20260925 19:04:50.513231 1419 webserver.cc:460] Webserver started at http://127.0.196.193:44003/ using document root <none> and password file <none>
I20260925 19:04:50.513468 1419 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:50.513537 1419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:50.513686 1419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:50.514863 1419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-0/data/instance:
uuid: "6d1ad49884114c6fafce3e83534a7ca8"
format_stamp: "Formatted at 2026-09-25 19:04:50 on dist-test-slave-rn98"
server_key: "93691d2e3083b2f443c28220249336a4"
server_key_iv: "0f526d38b0a474fd7ed0867574353196"
server_key_version: "encryptionkey@0"
I20260925 19:04:50.515218 1419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-0/wal/instance:
uuid: "6d1ad49884114c6fafce3e83534a7ca8"
format_stamp: "Formatted at 2026-09-25 19:04:50 on dist-test-slave-rn98"
server_key: "93691d2e3083b2f443c28220249336a4"
server_key_iv: "0f526d38b0a474fd7ed0867574353196"
server_key_version: "encryptionkey@0"
I20260925 19:04:50.516623 1419 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:04:50.517402 1434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:50.517580 1419 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:04:50.517683 1419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "6d1ad49884114c6fafce3e83534a7ca8"
format_stamp: "Formatted at 2026-09-25 19:04:50 on dist-test-slave-rn98"
server_key: "93691d2e3083b2f443c28220249336a4"
server_key_iv: "0f526d38b0a474fd7ed0867574353196"
server_key_version: "encryptionkey@0"
I20260925 19:04:50.517758 1419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:50.535115 1419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:50.537184 1419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:50.537358 1419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:50.537606 1419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:04:50.537948 1419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:50.538002 1419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:50.538043 1419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:50.538070 1419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:50.544320 1419 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:34465
I20260925 19:04:50.544363 1547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:34465 every 8 connection(s)
I20260925 19:04:50.544720 1419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
Sep 25 19:04:50 dist-test-slave-rn98 krb5kdc[1312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:04:50.552896 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 1419
I20260925 19:04:50.552968 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-0/wal/instance
I20260925 19:04:50.553153 787 external_mini_cluster.cc:1428] Setting key b94337041aa998de69e8a80a0eb91c8e
I20260925 19:04:50.554929 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:50.545546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:36623 (local address 127.0.196.254:42115)
0925 19:04:50.545620 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:50.545622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:50.545853 (+ 231us) server_negotiation.cc:407] Connection header received
0925 19:04:50.546334 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:50.546335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:50.546346 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:50.546403 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:50.547428 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.547926 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.548984 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.549198 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.549265 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:50.551734 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:50.551736 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:50.551736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:50.551743 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:50.553140 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:50.553676 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:50.553678 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:50.553679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:50.553748 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:50.554172 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:50.554173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:50.554174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:50.554290 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:50.554347 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:50.554782 (+ 435us) server_negotiation.cc:299] Negotiation successful
0925 19:04:50.554859 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 19:04:50.555270 1550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:50.545525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:42115 (local address 127.0.196.193:36623)
0925 19:04:50.545698 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:50.545739 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:50.546228 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:50.546466 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:50.546469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:50.547119 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:50.547321 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:50.547326 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.548026 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.548027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:50.548879 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:50.548882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.549274 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.549275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:50.549379 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:50.549872 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:50.549881 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:50.551632 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:50.553225 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:50.553233 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:50.553235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:50.553584 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:50.553934 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:50.553935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:50.553935 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:50.554095 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:50.554376 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:50.554383 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:50.554575 (+ 192us) client_negotiation.cc:769] Sending connection context
0925 19:04:50.554758 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 19:04:50.554959 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":96,"threads_started":1}
I20260925 19:04:50.555833 1548 heartbeater.cc:347] Connected to a master server at 127.0.196.254:42115
I20260925 19:04:50.555960 1548 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:50.556169 1548 heartbeater.cc:511] Master 127.0.196.254:42115 requested a full tablet report, sending...
I20260925 19:04:50.556841 1357 ts_manager.cc:194] Registered new tserver with Master: 6d1ad49884114c6fafce3e83534a7ca8 (127.0.196.193:34465)
I20260925 19:04:50.557803 1357 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.193@KRBTEST.COM'} at 127.0.196.193:36623
WARNING: no policy specified for kudu/127.0.196.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
I20260925 19:04:50.581890 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:0
--local_ip_for_outbound_sockets=127.0.196.194
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:42115
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:41555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:50.664678 1553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:50.664908 1553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:50.664934 1553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:50.666134 1553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:50.666186 1553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:50.666215 1553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:04:50.667757 1553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:41555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:42115
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:50.668040 1553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:50.668282 1553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:04:50.673619 1553 server_base.cc:1034] running on GCE node
W20260925 19:04:50.673578 1558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:50.673573 1561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:50.673565 1559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:50.674058 1553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:50.674278 1553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:50.675421 1553 hybrid_clock.cc:648] HybridClock initialized: now 1790363090675403 us; error 41 us; skew 500 ppm
Sep 25 19:04:50 dist-test-slave-rn98 krb5kdc[1312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:50.677934 1553 init.cc:377] Logged in from keytab as kudu/127.0.196.194@KRBTEST.COM (short username kudu)
I20260925 19:04:50.678334 1553 webserver.cc:460] Webserver started at http://127.0.196.194:38925/ using document root <none> and password file <none>
I20260925 19:04:50.678547 1553 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:50.678589 1553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:50.678757 1553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:50.679814 1553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-1/data/instance:
uuid: "e3f8abf88f7c4c6d8aea97f4306ab355"
format_stamp: "Formatted at 2026-09-25 19:04:50 on dist-test-slave-rn98"
server_key: "197f82736e3001e23c7754b2a170f685"
server_key_iv: "5aca6daf442ec2a2d824002ed08c368b"
server_key_version: "encryptionkey@0"
I20260925 19:04:50.680214 1553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-1/wal/instance:
uuid: "e3f8abf88f7c4c6d8aea97f4306ab355"
format_stamp: "Formatted at 2026-09-25 19:04:50 on dist-test-slave-rn98"
server_key: "197f82736e3001e23c7754b2a170f685"
server_key_iv: "5aca6daf442ec2a2d824002ed08c368b"
server_key_version: "encryptionkey@0"
I20260925 19:04:50.681687 1553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:04:50.682374 1568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:50.682531 1553 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:04:50.682655 1553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "e3f8abf88f7c4c6d8aea97f4306ab355"
format_stamp: "Formatted at 2026-09-25 19:04:50 on dist-test-slave-rn98"
server_key: "197f82736e3001e23c7754b2a170f685"
server_key_iv: "5aca6daf442ec2a2d824002ed08c368b"
server_key_version: "encryptionkey@0"
I20260925 19:04:50.682735 1553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:50.705587 1553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:50.707742 1553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:50.707867 1553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:50.708060 1553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:04:50.708391 1553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:50.708425 1553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:50.708449 1553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:50.708496 1553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:50.714241 1553 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:43963
I20260925 19:04:50.714378 1681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:43963 every 8 connection(s)
I20260925 19:04:50.714705 1553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
I20260925 19:04:50.717082 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 1553
I20260925 19:04:50.717154 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-1/wal/instance
I20260925 19:04:50.717334 787 external_mini_cluster.cc:1428] Setting key 3355a859441a2bc8165d7e988b5adcaf
Sep 25 19:04:50 dist-test-slave-rn98 krb5kdc[1312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:04:50.725443 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:50.715432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:51867 (local address 127.0.196.254:42115)
0925 19:04:50.715476 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:50.715479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:50.715674 (+ 195us) server_negotiation.cc:407] Connection header received
0925 19:04:50.716253 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:50.716255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:50.716267 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:50.716335 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:50.717395 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.718014 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.719131 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.719451 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.719496 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:50.722267 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:50.722277 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:50.722277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:50.722286 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:50.723982 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:50.724403 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:50.724404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:50.724404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:50.724470 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:50.724702 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:50.724703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:50.724704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:50.724899 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:50.724980 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:50.725225 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 19:04:50.725314 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:04:50.725468 1684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:50.715398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:42115 (local address 127.0.196.194:51867)
0925 19:04:50.715543 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:50.715584 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:50.716112 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:50.716409 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:50.716413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:50.717035 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:50.717298 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:50.717304 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.718091 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.718092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:50.719003 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:50.719005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.719588 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.719589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:50.719739 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:50.720390 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:50.720399 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:50.722170 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:50.724073 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:50.724080 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:50.724082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:50.724295 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:50.724541 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:50.724541 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:50.724541 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:50.724639 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:50.724970 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:50.724974 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:50.725092 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 19:04:50.725176 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 19:04:50.725295 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
I20260925 19:04:50.725967 1682 heartbeater.cc:347] Connected to a master server at 127.0.196.254:42115
I20260925 19:04:50.726078 1682 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:50.726263 1682 heartbeater.cc:511] Master 127.0.196.254:42115 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.196.195@KRBTEST.COM; defaulting to no policy
I20260925 19:04:50.726686 1357 ts_manager.cc:194] Registered new tserver with Master: e3f8abf88f7c4c6d8aea97f4306ab355 (127.0.196.194:43963)
I20260925 19:04:50.727070 1357 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.194@KRBTEST.COM'} at 127.0.196.194:51867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
I20260925 19:04:50.745476 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:0
--local_ip_for_outbound_sockets=127.0.196.195
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:42115
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:41555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:50.826965 1687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:50.827200 1687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:50.827225 1687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:50.828392 1687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:50.828433 1687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:50.828460 1687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:04:50.829866 1687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:41555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:42115
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:50.830063 1687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:50.830240 1687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:50.835603 1695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:50.835671 1693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:50.835834 1687 server_base.cc:1034] running on GCE node
W20260925 19:04:50.835614 1692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:50.836186 1687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:50.836447 1687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:50.837622 1687 hybrid_clock.cc:648] HybridClock initialized: now 1790363090837620 us; error 40 us; skew 500 ppm
Sep 25 19:04:50 dist-test-slave-rn98 krb5kdc[1312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:50.840389 1687 init.cc:377] Logged in from keytab as kudu/127.0.196.195@KRBTEST.COM (short username kudu)
I20260925 19:04:50.840891 1687 webserver.cc:460] Webserver started at http://127.0.196.195:40769/ using document root <none> and password file <none>
I20260925 19:04:50.841148 1687 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:50.841220 1687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:50.841374 1687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:50.842520 1687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-2/data/instance:
uuid: "897dd8bb3ebf400f9fb569cd7c5ba483"
format_stamp: "Formatted at 2026-09-25 19:04:50 on dist-test-slave-rn98"
server_key: "146032b07340f2fe451eb4d10b5cd131"
server_key_iv: "efaec4dc74802d8743f1dde622cab719"
server_key_version: "encryptionkey@0"
I20260925 19:04:50.842969 1687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-2/wal/instance:
uuid: "897dd8bb3ebf400f9fb569cd7c5ba483"
format_stamp: "Formatted at 2026-09-25 19:04:50 on dist-test-slave-rn98"
server_key: "146032b07340f2fe451eb4d10b5cd131"
server_key_iv: "efaec4dc74802d8743f1dde622cab719"
server_key_version: "encryptionkey@0"
I20260925 19:04:50.844406 1687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:04:50.845294 1702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:50.845510 1687 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:04:50.845605 1687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "897dd8bb3ebf400f9fb569cd7c5ba483"
format_stamp: "Formatted at 2026-09-25 19:04:50 on dist-test-slave-rn98"
server_key: "146032b07340f2fe451eb4d10b5cd131"
server_key_iv: "efaec4dc74802d8743f1dde622cab719"
server_key_version: "encryptionkey@0"
I20260925 19:04:50.845686 1687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:50.883249 1687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:50.885352 1687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:50.885526 1687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:50.885774 1687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:04:50.886118 1687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:50.886157 1687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:50.886205 1687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:50.886231 1687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:50.892633 1687 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:40461
I20260925 19:04:50.892692 1815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:40461 every 8 connection(s)
I20260925 19:04:50.893042 1687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
Sep 25 19:04:50 dist-test-slave-rn98 krb5kdc[1312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:04:50.900830 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 1687
I20260925 19:04:50.900952 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790363087086945-787-0/minicluster-data/ts-2/wal/instance
I20260925 19:04:50.901144 787 external_mini_cluster.cc:1428] Setting key 3e4a189a596ad8d46f349efb2176fb1b
I20260925 19:04:50.902998 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:50.893804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:35327 (local address 127.0.196.254:42115)
0925 19:04:50.893849 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:50.893851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:50.894045 (+ 194us) server_negotiation.cc:407] Connection header received
0925 19:04:50.894552 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:50.894553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:50.894564 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:50.894677 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:50.895633 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.896142 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.897195 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.897391 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.897455 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:50.899840 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:50.899841 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:50.899841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:50.899848 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:50.901430 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:50.901954 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:50.901955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:50.901956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:50.902016 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:50.902345 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:50.902347 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:50.902347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:50.902492 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:50.902557 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:50.902857 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 19:04:50.902927 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:04:50.903168 1818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:50.893792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:42115 (local address 127.0.196.195:35327)
0925 19:04:50.893934 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:50.893971 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:50.894442 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:50.894755 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:50.894758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:50.895339 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:50.895537 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:50.895542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.896305 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.896306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:50.897108 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:50.897110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.897495 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.897497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:50.897617 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:50.898163 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:50.898172 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:50.899750 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:50.901518 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:50.901525 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:50.901527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:50.901883 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:50.902091 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:50.902091 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:50.902092 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:50.902265 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:50.902550 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:50.902553 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:50.902735 (+ 182us) client_negotiation.cc:769] Sending connection context
0925 19:04:50.902824 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 19:04:50.902957 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
I20260925 19:04:50.903699 1816 heartbeater.cc:347] Connected to a master server at 127.0.196.254:42115
I20260925 19:04:50.903807 1816 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:50.903995 1816 heartbeater.cc:511] Master 127.0.196.254:42115 requested a full tablet report, sending...
I20260925 19:04:50.904393 1357 ts_manager.cc:194] Registered new tserver with Master: 897dd8bb3ebf400f9fb569cd7c5ba483 (127.0.196.195:40461)
I20260925 19:04:50.904819 1357 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.195@KRBTEST.COM'} at 127.0.196.195:35327
I20260925 19:04:50.913569 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:04:50.921731 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:50.915295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39816 (local address 127.0.196.254:42115)
0925 19:04:50.915362 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:50.915364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:50.915377 (+ 13us) server_negotiation.cc:407] Connection header received
0925 19:04:50.915471 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:50.915472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:50.915484 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:50.915542 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:50.916125 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.916670 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.917428 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.917626 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.917706 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:50.919050 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:50.919052 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:50.919052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:50.919060 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:50.920363 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:50.920706 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:50.920707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:50.920707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:50.920759 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:50.921092 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:50.921093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:50.921093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:50.921285 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:50.921412 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:50.921544 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 19:04:50.921654 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 19:04:50.923916 1357 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:04:50.925411 1357 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:04:50.932129 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:50.927907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:43963 (local address 127.0.0.1:40014)
0925 19:04:50.928211 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:50.928220 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:50.928270 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:50.928495 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:50.928499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:50.928502 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:50.928579 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:50.928581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.929997 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.929998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:50.931043 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:50.931046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.931895 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.931896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:50.931956 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:50.931968 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 19:04:50.932024 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 19:04:50.932063 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":96,"thread_start_us":96,"threads_started":1}
I20260925 19:04:50.932192 1829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:50.927982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40014 (local address 127.0.196.194:43963)
0925 19:04:50.928164 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:50.928169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:50.928338 (+ 169us) server_negotiation.cc:407] Connection header received
0925 19:04:50.928374 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:50.928377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:50.928398 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:50.928486 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:50.928662 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.929871 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.931139 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.931812 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.931898 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:50.931959 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:50.932061 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 19:04:50.932103 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":79,"threads_started":1}
I20260925 19:04:50.932811 1827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:50.927888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:40461 (local address 127.0.0.1:60224)
0925 19:04:50.928066 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:50.928107 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:50.928176 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:50.929021 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:50.929022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:50.929023 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:50.929087 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:50.929088 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.929949 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.929950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:50.931351 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:50.931354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.932567 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.932568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:50.932628 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:50.932631 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:04:50.932692 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 19:04:50.932734 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":62,"thread_start_us":67,"threads_started":1}
I20260925 19:04:50.932847 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:50.928004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60224 (local address 127.0.196.195:40461)
0925 19:04:50.928618 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:50.928622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:50.928631 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:04:50.928668 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:50.928670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:50.928684 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:50.928981 (+ 297us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:50.929188 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.929838 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.931440 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.932183 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.932258 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:50.932314 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:50.932729 (+ 415us) server_negotiation.cc:299] Negotiation successful
0925 19:04:50.932770 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":75,"threads_started":1}
I20260925 19:04:50.932972 1616 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc38818cc194c4b81c7d20374e1bc70 (DEFAULT_TABLE table=test-table [id=29a6d4855b594fdd8ce0a2fc77744acf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:50.933363 1616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc38818cc194c4b81c7d20374e1bc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:50.933408 1831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:50.928106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:34465 (local address 127.0.0.1:55208)
0925 19:04:50.928234 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:50.928245 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:50.928300 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:50.928728 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:50.928729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:50.928730 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:50.928771 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:50.928773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.930046 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.930047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:50.931351 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:50.931354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.933191 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.933192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:50.933256 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:50.933261 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:04:50.933293 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 19:04:50.933336 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":46,"thread_start_us":83,"threads_started":1}
I20260925 19:04:50.933784 1832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:50.928369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55208 (local address 127.0.196.193:34465)
0925 19:04:50.928562 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:50.928566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:50.928576 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:04:50.928615 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:50.928633 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:50.928649 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:50.928735 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:50.928847 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.929903 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.932356 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.933084 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.933568 (+ 484us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:50.933636 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:50.933662 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 19:04:50.933704 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":110,"threads_started":1}
I20260925 19:04:50.933823 1750 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc38818cc194c4b81c7d20374e1bc70 (DEFAULT_TABLE table=test-table [id=29a6d4855b594fdd8ce0a2fc77744acf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:50.934159 1750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc38818cc194c4b81c7d20374e1bc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:50.934368 1482 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc38818cc194c4b81c7d20374e1bc70 (DEFAULT_TABLE table=test-table [id=29a6d4855b594fdd8ce0a2fc77744acf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:50.934710 1482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc38818cc194c4b81c7d20374e1bc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:50.936532 1833 tablet_bootstrap.cc:492] T 3fc38818cc194c4b81c7d20374e1bc70 P e3f8abf88f7c4c6d8aea97f4306ab355: Bootstrap starting.
I20260925 19:04:50.937522 1833 tablet_bootstrap.cc:654] T 3fc38818cc194c4b81c7d20374e1bc70 P e3f8abf88f7c4c6d8aea97f4306ab355: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:50.937604 1834 tablet_bootstrap.cc:492] T 3fc38818cc194c4b81c7d20374e1bc70 P 897dd8bb3ebf400f9fb569cd7c5ba483: Bootstrap starting.
I20260925 19:04:50.938060 1835 tablet_bootstrap.cc:492] T 3fc38818cc194c4b81c7d20374e1bc70 P 6d1ad49884114c6fafce3e83534a7ca8: Bootstrap starting.
I20260925 19:04:50.938701 1835 tablet_bootstrap.cc:654] T 3fc38818cc194c4b81c7d20374e1bc70 P 6d1ad49884114c6fafce3e83534a7ca8: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:50.938812 1833 log.cc:824] T 3fc38818cc194c4b81c7d20374e1bc70 P e3f8abf88f7c4c6d8aea97f4306ab355: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:50.938889 1834 tablet_bootstrap.cc:654] T 3fc38818cc194c4b81c7d20374e1bc70 P 897dd8bb3ebf400f9fb569cd7c5ba483: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:50.939329 1835 log.cc:824] T 3fc38818cc194c4b81c7d20374e1bc70 P 6d1ad49884114c6fafce3e83534a7ca8: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:50.939399 1834 log.cc:824] T 3fc38818cc194c4b81c7d20374e1bc70 P 897dd8bb3ebf400f9fb569cd7c5ba483: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:50.939709 1833 tablet_bootstrap.cc:492] T 3fc38818cc194c4b81c7d20374e1bc70 P e3f8abf88f7c4c6d8aea97f4306ab355: No bootstrap required, opened a new log
I20260925 19:04:50.939807 1833 ts_tablet_manager.cc:1397] T 3fc38818cc194c4b81c7d20374e1bc70 P e3f8abf88f7c4c6d8aea97f4306ab355: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 19:04:50.940053 1835 tablet_bootstrap.cc:492] T 3fc38818cc194c4b81c7d20374e1bc70 P 6d1ad49884114c6fafce3e83534a7ca8: No bootstrap required, opened a new log
I20260925 19:04:50.940128 1834 tablet_bootstrap.cc:492] T 3fc38818cc194c4b81c7d20374e1bc70 P 897dd8bb3ebf400f9fb569cd7c5ba483: No bootstrap required, opened a new log
I20260925 19:04:50.940138 1835 ts_tablet_manager.cc:1397] T 3fc38818cc194c4b81c7d20374e1bc70 P 6d1ad49884114c6fafce3e83534a7ca8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 19:04:50.940200 1834 ts_tablet_manager.cc:1397] T 3fc38818cc194c4b81c7d20374e1bc70 P 897dd8bb3ebf400f9fb569cd7c5ba483: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 19:04:50.941591 1835 raft_consensus.cc:348] T 3fc38818cc194c4b81c7d20374e1bc70 P 6d1ad49884114c6fafce3e83534a7ca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "897dd8bb3ebf400f9fb569cd7c5ba483" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 40461 } } peers { permanent_uuid: "e3f8abf88f7c4c6d8aea97f4306ab355" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43963 } } peers { permanent_uuid: "6d1ad49884114c6fafce3e83534a7ca8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34465 } }
I20260925 19:04:50.941766 1835 raft_consensus.cc:374] T 3fc38818cc194c4b81c7d20374e1bc70 P 6d1ad49884114c6fafce3e83534a7ca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:50.941811 1835 raft_consensus.cc:729] T 3fc38818cc194c4b81c7d20374e1bc70 P 6d1ad49884114c6fafce3e83534a7ca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1ad49884114c6fafce3e83534a7ca8, State: Initialized, Role: FOLLOWER
I20260925 19:04:50.941936 1835 consensus_queue.cc:260] T 3fc38818cc194c4b81c7d20374e1bc70 P 6d1ad49884114c6fafce3e83534a7ca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "897dd8bb3ebf400f9fb569cd7c5ba483" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 40461 } } peers { permanent_uuid: "e3f8abf88f7c4c6d8aea97f4306ab355" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43963 } } peers { permanent_uuid: "6d1ad49884114c6fafce3e83534a7ca8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34465 } }
I20260925 19:04:50.941896 1833 raft_consensus.cc:348] T 3fc38818cc194c4b81c7d20374e1bc70 P e3f8abf88f7c4c6d8aea97f4306ab355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "897dd8bb3ebf400f9fb569cd7c5ba483" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 40461 } } peers { permanent_uuid: "e3f8abf88f7c4c6d8aea97f4306ab355" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43963 } } peers { permanent_uuid: "6d1ad49884114c6fafce3e83534a7ca8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34465 } }
I20260925 19:04:50.942085 1833 raft_consensus.cc:374] T 3fc38818cc194c4b81c7d20374e1bc70 P e3f8abf88f7c4c6d8aea97f4306ab355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:50.942119 1833 raft_consensus.cc:729] T 3fc38818cc194c4b81c7d20374e1bc70 P e3f8abf88f7c4c6d8aea97f4306ab355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f8abf88f7c4c6d8aea97f4306ab355, State: Initialized, Role: FOLLOWER
I20260925 19:04:50.942252 1835 ts_tablet_manager.cc:1428] T 3fc38818cc194c4b81c7d20374e1bc70 P 6d1ad49884114c6fafce3e83534a7ca8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 19:04:50.942245 1833 consensus_queue.cc:260] T 3fc38818cc194c4b81c7d20374e1bc70 P e3f8abf88f7c4c6d8aea97f4306ab355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "897dd8bb3ebf400f9fb569cd7c5ba483" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 40461 } } peers { permanent_uuid: "e3f8abf88f7c4c6d8aea97f4306ab355" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43963 } } peers { permanent_uuid: "6d1ad49884114c6fafce3e83534a7ca8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34465 } }
I20260925 19:04:50.942384 1548 heartbeater.cc:503] Master 127.0.196.254:42115 was elected leader, sending a full tablet report...
I20260925 19:04:50.942535 1682 heartbeater.cc:503] Master 127.0.196.254:42115 was elected leader, sending a full tablet report...
I20260925 19:04:50.942665 1834 raft_consensus.cc:348] T 3fc38818cc194c4b81c7d20374e1bc70 P 897dd8bb3ebf400f9fb569cd7c5ba483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "897dd8bb3ebf400f9fb569cd7c5ba483" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 40461 } } peers { permanent_uuid: "e3f8abf88f7c4c6d8aea97f4306ab355" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43963 } } peers { permanent_uuid: "6d1ad49884114c6fafce3e83534a7ca8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34465 } }
I20260925 19:04:50.942523 1833 ts_tablet_manager.cc:1428] T 3fc38818cc194c4b81c7d20374e1bc70 P e3f8abf88f7c4c6d8aea97f4306ab355: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:04:50.942836 1834 raft_consensus.cc:374] T 3fc38818cc194c4b81c7d20374e1bc70 P 897dd8bb3ebf400f9fb569cd7c5ba483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:50.942898 1834 raft_consensus.cc:729] T 3fc38818cc194c4b81c7d20374e1bc70 P 897dd8bb3ebf400f9fb569cd7c5ba483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 897dd8bb3ebf400f9fb569cd7c5ba483, State: Initialized, Role: FOLLOWER
I20260925 19:04:50.943010 1834 consensus_queue.cc:260] T 3fc38818cc194c4b81c7d20374e1bc70 P 897dd8bb3ebf400f9fb569cd7c5ba483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "897dd8bb3ebf400f9fb569cd7c5ba483" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 40461 } } peers { permanent_uuid: "e3f8abf88f7c4c6d8aea97f4306ab355" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43963 } } peers { permanent_uuid: "6d1ad49884114c6fafce3e83534a7ca8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34465 } }
I20260925 19:04:50.943298 1816 heartbeater.cc:503] Master 127.0.196.254:42115 was elected leader, sending a full tablet report...
I20260925 19:04:50.943305 1834 ts_tablet_manager.cc:1428] T 3fc38818cc194c4b81c7d20374e1bc70 P 897dd8bb3ebf400f9fb569cd7c5ba483: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260925 19:04:50.965262 1683 tablet.cc:2369] T 3fc38818cc194c4b81c7d20374e1bc70 P e3f8abf88f7c4c6d8aea97f4306ab355: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:04:50.975850 1839 raft_consensus.cc:482] T 3fc38818cc194c4b81c7d20374e1bc70 P 6d1ad49884114c6fafce3e83534a7ca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:04:50.975973 1839 raft_consensus.cc:504] T 3fc38818cc194c4b81c7d20374e1bc70 P 6d1ad49884114c6fafce3e83534a7ca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "897dd8bb3ebf400f9fb569cd7c5ba483" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 40461 } } peers { permanent_uuid: "e3f8abf88f7c4c6d8aea97f4306ab355" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43963 } } peers { permanent_uuid: "6d1ad49884114c6fafce3e83534a7ca8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34465 } }
I20260925 19:04:50.976456 1839 leader_election.cc:288] T 3fc38818cc194c4b81c7d20374e1bc70 P 6d1ad49884114c6fafce3e83534a7ca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 897dd8bb3ebf400f9fb569cd7c5ba483 (127.0.196.195:40461), e3f8abf88f7c4c6d8aea97f4306ab355 (127.0.196.194:43963)
I20260925 19:04:50.979245 1550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:50.976609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:40461 (local address 127.0.196.193:38179)
0925 19:04:50.976659 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:50.976671 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:50.976713 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:50.976856 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:50.976858 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:50.976859 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:50.976936 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:50.976938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.977580 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.977581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:50.978376 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:50.978378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.978974 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.978975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:50.979054 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:50.979058 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:04:50.979123 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 19:04:50.979164 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 19:04:50.979245 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:50.976664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:38179 (local address 127.0.196.195:40461)
0925 19:04:50.976733 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:50.976735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:50.976761 (+ 26us) server_negotiation.cc:407] Connection header received
0925 19:04:50.976792 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:50.976793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:50.976803 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:50.976852 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:50.977014 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.977492 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.978478 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.978885 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.978977 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:50.978988 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:50.979142 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 19:04:50.979181 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 19:04:50.979633 1770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc38818cc194c4b81c7d20374e1bc70" candidate_uuid: "6d1ad49884114c6fafce3e83534a7ca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "897dd8bb3ebf400f9fb569cd7c5ba483" is_pre_election: true
I20260925 19:04:50.979851 1770 raft_consensus.cc:2454] T 3fc38818cc194c4b81c7d20374e1bc70 P 897dd8bb3ebf400f9fb569cd7c5ba483 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1ad49884114c6fafce3e83534a7ca8 in term 0.
I20260925 19:04:50.979919 1829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:50.976844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:46891 (local address 127.0.196.194:43963)
0925 19:04:50.976894 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:50.976896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:50.976905 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:04:50.976995 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:50.976996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:50.977007 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:50.977074 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:50.977227 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.977675 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.979062 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.979576 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.979653 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:50.979662 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:50.979810 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 19:04:50.979854 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 19:04:50.979943 1842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:50.976643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:43963 (local address 127.0.196.193:46891)
0925 19:04:50.976853 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:50.976864 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:50.976913 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:50.977077 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:50.977078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:50.977079 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:50.977156 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:50.977158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.977760 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.977762 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:50.978967 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:50.978969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.979659 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.979660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:50.979714 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:50.979718 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:04:50.979794 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 19:04:50.979835 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":138,"threads_started":1}
I20260925 19:04:50.980146 1438 leader_election.cc:302] T 3fc38818cc194c4b81c7d20374e1bc70 P 6d1ad49884114c6fafce3e83534a7ca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1ad49884114c6fafce3e83534a7ca8, 897dd8bb3ebf400f9fb569cd7c5ba483; no voters:
I20260925 19:04:50.980309 1636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc38818cc194c4b81c7d20374e1bc70" candidate_uuid: "6d1ad49884114c6fafce3e83534a7ca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3f8abf88f7c4c6d8aea97f4306ab355" is_pre_election: true
I20260925 19:04:50.980432 1636 raft_consensus.cc:2454] T 3fc38818cc194c4b81c7d20374e1bc70 P e3f8abf88f7c4c6d8aea97f4306ab355 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1ad49884114c6fafce3e83534a7ca8 in term 0.
I20260925 19:04:50.980546 1839 raft_consensus.cc:2781] T 3fc38818cc194c4b81c7d20374e1bc70 P 6d1ad49884114c6fafce3e83534a7ca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:04:50.980621 1839 raft_consensus.cc:482] T 3fc38818cc194c4b81c7d20374e1bc70 P 6d1ad49884114c6fafce3e83534a7ca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:04:50.980672 1839 raft_consensus.cc:3037] T 3fc38818cc194c4b81c7d20374e1bc70 P 6d1ad49884114c6fafce3e83534a7ca8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:50.981412 1839 raft_consensus.cc:504] T 3fc38818cc194c4b81c7d20374e1bc70 P 6d1ad49884114c6fafce3e83534a7ca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "897dd8bb3ebf400f9fb569cd7c5ba483" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 40461 } } peers { permanent_uuid: "e3f8abf88f7c4c6d8aea97f4306ab355" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43963 } } peers { permanent_uuid: "6d1ad49884114c6fafce3e83534a7ca8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34465 } }
I20260925 19:04:50.981600 1839 leader_election.cc:288] T 3fc38818cc194c4b81c7d20374e1bc70 P 6d1ad49884114c6fafce3e83534a7ca8 [CANDIDATE]: Term 1 election: Requested vote from peers 897dd8bb3ebf400f9fb569cd7c5ba483 (127.0.196.195:40461), e3f8abf88f7c4c6d8aea97f4306ab355 (127.0.196.194:43963)
I20260925 19:04:50.981699 1770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc38818cc194c4b81c7d20374e1bc70" candidate_uuid: "6d1ad49884114c6fafce3e83534a7ca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "897dd8bb3ebf400f9fb569cd7c5ba483"
I20260925 19:04:50.981786 1770 raft_consensus.cc:3037] T 3fc38818cc194c4b81c7d20374e1bc70 P 897dd8bb3ebf400f9fb569cd7c5ba483 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:50.981799 1636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc38818cc194c4b81c7d20374e1bc70" candidate_uuid: "6d1ad49884114c6fafce3e83534a7ca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3f8abf88f7c4c6d8aea97f4306ab355"
I20260925 19:04:50.981921 1636 raft_consensus.cc:3037] T 3fc38818cc194c4b81c7d20374e1bc70 P e3f8abf88f7c4c6d8aea97f4306ab355 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:50.982553 1770 raft_consensus.cc:2454] T 3fc38818cc194c4b81c7d20374e1bc70 P 897dd8bb3ebf400f9fb569cd7c5ba483 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1ad49884114c6fafce3e83534a7ca8 in term 1.
I20260925 19:04:50.982761 1438 leader_election.cc:302] T 3fc38818cc194c4b81c7d20374e1bc70 P 6d1ad49884114c6fafce3e83534a7ca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1ad49884114c6fafce3e83534a7ca8, 897dd8bb3ebf400f9fb569cd7c5ba483; no voters:
I20260925 19:04:50.982782 1636 raft_consensus.cc:2454] T 3fc38818cc194c4b81c7d20374e1bc70 P e3f8abf88f7c4c6d8aea97f4306ab355 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1ad49884114c6fafce3e83534a7ca8 in term 1.
I20260925 19:04:50.982869 1839 raft_consensus.cc:2781] T 3fc38818cc194c4b81c7d20374e1bc70 P 6d1ad49884114c6fafce3e83534a7ca8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:50.983040 1839 raft_consensus.cc:686] T 3fc38818cc194c4b81c7d20374e1bc70 P 6d1ad49884114c6fafce3e83534a7ca8 [term 1 LEADER]: Becoming Leader. State: Replica: 6d1ad49884114c6fafce3e83534a7ca8, State: Running, Role: LEADER
I20260925 19:04:50.983426 1839 consensus_queue.cc:237] T 3fc38818cc194c4b81c7d20374e1bc70 P 6d1ad49884114c6fafce3e83534a7ca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "897dd8bb3ebf400f9fb569cd7c5ba483" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 40461 } } peers { permanent_uuid: "e3f8abf88f7c4c6d8aea97f4306ab355" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43963 } } peers { permanent_uuid: "6d1ad49884114c6fafce3e83534a7ca8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34465 } }
I20260925 19:04:50.984416 1355 catalog_manager.cc:5495] T 3fc38818cc194c4b81c7d20374e1bc70 P 6d1ad49884114c6fafce3e83534a7ca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d1ad49884114c6fafce3e83534a7ca8 (127.0.196.193). New cstate: current_term: 1 leader_uuid: "6d1ad49884114c6fafce3e83534a7ca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "897dd8bb3ebf400f9fb569cd7c5ba483" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3f8abf88f7c4c6d8aea97f4306ab355" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d1ad49884114c6fafce3e83534a7ca8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34465 } health_report { overall_health: HEALTHY } } }
Sep 25 19:04:50 dist-test-slave-rn98 krb5kdc[1312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.193@KRBTEST.COM
I20260925 19:04:50.997368 1832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:50.988129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55210 (local address 127.0.196.193:34465)
0925 19:04:50.988174 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:50.988176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:50.988328 (+ 152us) server_negotiation.cc:407] Connection header received
0925 19:04:50.988371 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:50.988373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:50.988399 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:50.988455 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:50.989281 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.990125 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.991084 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:50.991375 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:50.991466 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:50.994007 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:50.994021 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:50.994021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:50.994029 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:50.995930 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:50.996450 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:50.996454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:50.996455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:50.996528 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:50.996747 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:50.996749 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:50.996749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:50.996920 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:50.997119 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:50.997160 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 19:04:50.997250 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 25 19:04:51 dist-test-slave-rn98 krb5kdc[1312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363091, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 19:04:51 dist-test-slave-rn98 krb5kdc[1312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363091, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.196.193@KRBTEST.COM
I20260925 19:04:51.021652 1832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:51.013655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55224 (local address 127.0.196.193:34465)
0925 19:04:51.013727 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:51.013729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:51.013774 (+ 45us) server_negotiation.cc:407] Connection header received
0925 19:04:51.013869 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:51.013870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:51.013879 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:51.013952 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:51.014591 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:51.015215 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:51.015943 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:51.016137 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:51.016207 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:51.018973 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:51.018975 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:51.018975 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:51.018983 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:51.020303 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:51.020684 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:51.020686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:51.020686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:51.020768 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:51.020974 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:51.020975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:51.020976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:51.021105 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:51.021204 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:51.021509 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 19:04:51.021584 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 19:04:51 dist-test-slave-rn98 krb5kdc[1312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 19:04:51 dist-test-slave-rn98 krb5kdc[1312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.193@KRBTEST.COM
W20260925 19:04:51.045477 1549 tablet.cc:2369] T 3fc38818cc194c4b81c7d20374e1bc70 P 6d1ad49884114c6fafce3e83534a7ca8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:04:51.046187 1832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:51.038878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55226 (local address 127.0.196.193:34465)
0925 19:04:51.038951 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:51.038954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:51.039070 (+ 116us) server_negotiation.cc:407] Connection header received
0925 19:04:51.039160 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:51.039161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:51.039172 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:51.039230 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:51.039822 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:51.040314 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:51.041068 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:51.041268 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:51.041345 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:51.043787 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:51.043789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:51.043789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:51.043796 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:51.045039 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:51.045411 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:51.045412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:51.045412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:51.045472 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:51.045695 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:51.045696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:51.045696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:51.045827 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:51.045910 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:51.046030 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 19:04:51.046093 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 19:04:51.046761 1462 tablet_service.cc:2908] Rejecting scan request for tablet 3fc38818cc194c4b81c7d20374e1bc70: Uninitialized: clean time has not yet been initialized
I20260925 19:04:51.048103 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 1419
I20260925 19:04:51.053280 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 1553
I20260925 19:04:51.057762 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 1687
I20260925 19:04:51.062220 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 1325
2026-09-25T19:04:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2080 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:04:51 dist-test-slave-rn98 krb5kdc[1857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:04:51 dist-test-slave-rn98 krb5kdc[1857](info): set up 2 sockets
Sep 25 19:04:51 dist-test-slave-rn98 krb5kdc[1857](info): commencing operation
krb5kdc: starting...
W20260925 19:04:52.106247 787 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:04:52 dist-test-slave-rn98 krb5kdc[1857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:04:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:04:52.230186 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:44651
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:44651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:52.312119 1870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:52.312356 1870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:52.312381 1870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:52.313557 1870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:52.313601 1870 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 19:04:52.313616 1870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:52.313628 1870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:52.313640 1870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:52.313652 1870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:52.315171 1870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:44651
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:44651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:52.315380 1870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:52.315551 1870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:52.320624 1875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:52.320643 1878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:52.320624 1876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:52.321152 1870 server_base.cc:1034] running on GCE node
I20260925 19:04:52.321463 1870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:52.321794 1870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:52.322963 1870 hybrid_clock.cc:648] HybridClock initialized: now 1790363092322956 us; error 38 us; skew 500 ppm
Sep 25 19:04:52 dist-test-slave-rn98 krb5kdc[1857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:52.332065 1870 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:04:52.332722 1870 webserver.cc:460] Webserver started at http://127.0.196.254:42659/ using document root <none> and password file <none>
I20260925 19:04:52.332990 1870 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:52.333072 1870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:52.333243 1870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:52.334503 1870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "4fede94c0b9b47c3a3b92e5d8df187cb"
format_stamp: "Formatted at 2026-09-25 19:04:52 on dist-test-slave-rn98"
server_key: "ad912d5361c5f41fdef073481c9c6fff"
server_key_iv: "e06620eda132fccebdffa924b5a1c3ba"
server_key_version: "encryptionkey@0"
I20260925 19:04:52.335001 1870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "4fede94c0b9b47c3a3b92e5d8df187cb"
format_stamp: "Formatted at 2026-09-25 19:04:52 on dist-test-slave-rn98"
server_key: "ad912d5361c5f41fdef073481c9c6fff"
server_key_iv: "e06620eda132fccebdffa924b5a1c3ba"
server_key_version: "encryptionkey@0"
I20260925 19:04:52.336668 1870 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:04:52.337505 1885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:52.337708 1870 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:04:52.337822 1870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "4fede94c0b9b47c3a3b92e5d8df187cb"
format_stamp: "Formatted at 2026-09-25 19:04:52 on dist-test-slave-rn98"
server_key: "ad912d5361c5f41fdef073481c9c6fff"
server_key_iv: "e06620eda132fccebdffa924b5a1c3ba"
server_key_version: "encryptionkey@0"
I20260925 19:04:52.337918 1870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:52.348228 1870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:52.350275 1870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:52.350461 1870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:52.355803 1870 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:44651
I20260925 19:04:52.355842 1947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:44651 every 8 connection(s)
I20260925 19:04:52.356220 1870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:52.356853 1948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:52.359305 1948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fede94c0b9b47c3a3b92e5d8df187cb: Bootstrap starting.
I20260925 19:04:52.359935 1948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fede94c0b9b47c3a3b92e5d8df187cb: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:52.360240 1948 log.cc:824] T 00000000000000000000000000000000 P 4fede94c0b9b47c3a3b92e5d8df187cb: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:52.360985 1948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fede94c0b9b47c3a3b92e5d8df187cb: No bootstrap required, opened a new log
I20260925 19:04:52.363166 1948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fede94c0b9b47c3a3b92e5d8df187cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fede94c0b9b47c3a3b92e5d8df187cb" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 44651 } }
I20260925 19:04:52.363318 1948 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fede94c0b9b47c3a3b92e5d8df187cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:52.363348 1948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fede94c0b9b47c3a3b92e5d8df187cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fede94c0b9b47c3a3b92e5d8df187cb, State: Initialized, Role: FOLLOWER
I20260925 19:04:52.363538 1948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fede94c0b9b47c3a3b92e5d8df187cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fede94c0b9b47c3a3b92e5d8df187cb" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 44651 } }
I20260925 19:04:52.363605 1948 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fede94c0b9b47c3a3b92e5d8df187cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:04:52.363647 1948 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fede94c0b9b47c3a3b92e5d8df187cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:04:52.363699 1948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fede94c0b9b47c3a3b92e5d8df187cb [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:52.364301 1948 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fede94c0b9b47c3a3b92e5d8df187cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fede94c0b9b47c3a3b92e5d8df187cb" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 44651 } }
I20260925 19:04:52.364399 1948 leader_election.cc:302] T 00000000000000000000000000000000 P 4fede94c0b9b47c3a3b92e5d8df187cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fede94c0b9b47c3a3b92e5d8df187cb; no voters:
I20260925 19:04:52.364599 1948 leader_election.cc:288] T 00000000000000000000000000000000 P 4fede94c0b9b47c3a3b92e5d8df187cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:04:52.364655 1951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fede94c0b9b47c3a3b92e5d8df187cb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:52.364884 1948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fede94c0b9b47c3a3b92e5d8df187cb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:52.364907 1951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fede94c0b9b47c3a3b92e5d8df187cb [term 1 LEADER]: Becoming Leader. State: Replica: 4fede94c0b9b47c3a3b92e5d8df187cb, State: Running, Role: LEADER
I20260925 19:04:52.364997 1951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fede94c0b9b47c3a3b92e5d8df187cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fede94c0b9b47c3a3b92e5d8df187cb" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 44651 } }
I20260925 19:04:52.365216 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 1870
I20260925 19:04:52.365298 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:04:52.365388 1952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fede94c0b9b47c3a3b92e5d8df187cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fede94c0b9b47c3a3b92e5d8df187cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fede94c0b9b47c3a3b92e5d8df187cb" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 44651 } } }
I20260925 19:04:52.365417 1953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fede94c0b9b47c3a3b92e5d8df187cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fede94c0b9b47c3a3b92e5d8df187cb. Latest consensus state: current_term: 1 leader_uuid: "4fede94c0b9b47c3a3b92e5d8df187cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fede94c0b9b47c3a3b92e5d8df187cb" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 44651 } } }
I20260925 19:04:52.365460 1952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fede94c0b9b47c3a3b92e5d8df187cb [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:52.365487 1953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fede94c0b9b47c3a3b92e5d8df187cb [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:52.365490 787 external_mini_cluster.cc:1428] Setting key 87bb07794befde35f4da596236b645d5
I20260925 19:04:52.365662 1955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:04:52.366336 1955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:52.369813 1955 catalog_manager.cc:1347] Generated new cluster ID: 130cab300b4147dd84678712a93dcf3d
I20260925 19:04:52.369879 1955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:04:52 dist-test-slave-rn98 krb5kdc[1857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:04:52.376739 1963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:52.366160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.0.196.254:44651)
0925 19:04:52.366520 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:52.366532 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:52.366569 (+ 37us) server_negotiation.cc:407] Connection header received
0925 19:04:52.366748 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:52.366757 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:52.366828 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:52.367854 (+ 1026us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:52.367963 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.369319 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.370242 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.370811 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.370952 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:52.373319 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:52.373323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:52.373324 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:52.373349 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:52.375127 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:52.375508 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:52.375511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:52.375512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:52.375617 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:52.375830 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:52.375831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:52.375831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:52.376086 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:52.376275 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:52.376382 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 19:04:52.376528 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":95,"threads_started":1}
I20260925 19:04:52.385896 1955 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:52.386482 1955 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:52.397055 1955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fede94c0b9b47c3a3b92e5d8df187cb: Generated new TSK 0
I20260925 19:04:52.397285 1955 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
I20260925 19:04:52.434674 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:0
--local_ip_for_outbound_sockets=127.0.196.193
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:44651
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:52.537416 1974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:52.537642 1974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:52.537668 1974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:52.538831 1974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:52.538872 1974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:52.538892 1974 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 19:04:52.538908 1974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:04:52.540452 1974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.196.254:44651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:52.540659 1974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:52.540848 1974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:52.546069 1979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:52.546201 1982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:52.546204 1980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:52.546537 1974 server_base.cc:1034] running on GCE node
I20260925 19:04:52.546803 1974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:52.547070 1974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:52.548220 1974 hybrid_clock.cc:648] HybridClock initialized: now 1790363092548190 us; error 46 us; skew 500 ppm
Sep 25 19:04:52 dist-test-slave-rn98 krb5kdc[1857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:52.550967 1974 init.cc:377] Logged in from keytab as kudu/127.0.196.193@KRBTEST.COM (short username kudu)
I20260925 19:04:52.551407 1974 webserver.cc:460] Webserver started at http://127.0.196.193:45343/ using document root <none> and password file <none>
I20260925 19:04:52.551601 1974 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:52.551642 1974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:52.551731 1974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:52.552774 1974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-0/data/instance:
uuid: "3db90292012d4a61904ea62ffb27dc8f"
format_stamp: "Formatted at 2026-09-25 19:04:52 on dist-test-slave-rn98"
server_key: "aa8d400303071936cc4182b76e6abaa1"
server_key_iv: "d2c9a94e21ecf871217956756eb30969"
server_key_version: "encryptionkey@0"
I20260925 19:04:52.553077 1974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-0/wal/instance:
uuid: "3db90292012d4a61904ea62ffb27dc8f"
format_stamp: "Formatted at 2026-09-25 19:04:52 on dist-test-slave-rn98"
server_key: "aa8d400303071936cc4182b76e6abaa1"
server_key_iv: "d2c9a94e21ecf871217956756eb30969"
server_key_version: "encryptionkey@0"
I20260925 19:04:52.554363 1974 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 19:04:52.555100 1989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:52.555263 1974 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:52.555375 1974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "3db90292012d4a61904ea62ffb27dc8f"
format_stamp: "Formatted at 2026-09-25 19:04:52 on dist-test-slave-rn98"
server_key: "aa8d400303071936cc4182b76e6abaa1"
server_key_iv: "d2c9a94e21ecf871217956756eb30969"
server_key_version: "encryptionkey@0"
I20260925 19:04:52.555454 1974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:52.576851 1974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:52.579411 1974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:52.579591 1974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:52.580137 1974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:52.580204 1974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:52.580251 1974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:52.580278 1974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 19:04:52 dist-test-slave-rn98 krb5kdc[1857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:04:52.587568 1974 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:36747
I20260925 19:04:52.587630 2104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:36747 every 8 connection(s)
I20260925 19:04:52.588034 1974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
I20260925 19:04:52.590607 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 1974
I20260925 19:04:52.590771 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-0/wal/instance
I20260925 19:04:52.590945 787 external_mini_cluster.cc:1428] Setting key 80a76a29292d331ce66ba89d4440908b
I20260925 19:04:52.591218 1998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:52.580754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:44651 (local address 127.0.196.193:46565)
0925 19:04:52.580929 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:52.580968 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:52.581520 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:52.581845 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:52.581850 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:52.582277 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:52.582587 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:52.582594 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.583743 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.583744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:52.584658 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:52.584660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.585285 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.585286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:52.585438 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:52.586103 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:52.586114 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:52.587744 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:52.589617 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:52.589625 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:52.589627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:52.589865 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:52.590079 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:52.590080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:52.590080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:52.590194 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:52.590472 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:52.590478 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:52.590664 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 19:04:52.590782 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 19:04:52.590946 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":93,"threads_started":1}
I20260925 19:04:52.591598 1963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:52.580779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:46565 (local address 127.0.196.254:44651)
0925 19:04:52.580851 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:52.580853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:52.581069 (+ 216us) server_negotiation.cc:407] Connection header received
0925 19:04:52.581662 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:52.581664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:52.581682 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:52.581765 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:52.582723 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.583631 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.584775 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.585175 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.585261 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:52.587876 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:52.587877 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:52.587878 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:52.587887 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:52.589507 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:52.589945 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:52.589947 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:52.589947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:52.590012 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:52.590262 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:52.590263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:52.590263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:52.590391 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:52.590471 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:52.591389 (+ 918us) server_negotiation.cc:299] Negotiation successful
0925 19:04:52.591504 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 19:04:52.593238 2105 heartbeater.cc:347] Connected to a master server at 127.0.196.254:44651
I20260925 19:04:52.593390 2105 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:52.593612 2105 heartbeater.cc:511] Master 127.0.196.254:44651 requested a full tablet report, sending...
I20260925 19:04:52.594228 1901 ts_manager.cc:194] Registered new tserver with Master: 3db90292012d4a61904ea62ffb27dc8f (127.0.196.193:36747)
I20260925 19:04:52.595141 1901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.193@KRBTEST.COM'} at 127.0.196.193:46565
WARNING: no policy specified for kudu/127.0.196.194@KRBTEST.COM; defaulting to no policy
I20260925 19:04:52.600914 1963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:52.594314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:33237 (local address 127.0.196.254:44651)
0925 19:04:52.594353 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:52.594355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:52.594460 (+ 105us) server_negotiation.cc:407] Connection header received
0925 19:04:52.594506 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:52.594507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:52.594517 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:52.594572 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:52.594963 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.595756 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.596520 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.596879 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.596966 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:52.598036 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:52.598038 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:52.598038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:52.598047 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:52.599670 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:52.600083 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:52.600085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:52.600085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:52.600150 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:52.600360 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:52.600361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:52.600361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:52.600477 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:52.600550 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:52.600723 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 19:04:52.600811 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:04:52.600924 2112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:52.594245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:44651 (local address 127.0.196.193:33237)
0925 19:04:52.594383 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:52.594395 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:52.594457 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:52.594586 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:52.594587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:52.594811 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:52.594884 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:52.594885 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.595860 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.595861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:52.596431 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:52.596433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.596983 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.596984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:52.597045 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:52.597603 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:52.597608 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:52.597941 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:52.599773 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:52.599774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:52.599774 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:52.600014 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:52.600219 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:52.600220 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:52.600220 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:52.600281 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:52.600552 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:52.600553 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:52.600612 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 19:04:52.600698 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 19:04:52.600811 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
I20260925 19:04:52.617655 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:0
--local_ip_for_outbound_sockets=127.0.196.194
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:44651
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:52.700747 2114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:52.700971 2114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:52.700997 2114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:52.702193 2114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:52.702240 2114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:52.702261 2114 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 19:04:52.702279 2114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:04:52.703838 2114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.196.254:44651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:52.704092 2114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:52.704413 2114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:52.709546 2119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:52.709585 2114 server_base.cc:1034] running on GCE node
W20260925 19:04:52.709563 2120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:52.709566 2122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:52.709949 2114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:52.710162 2114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:52.711298 2114 hybrid_clock.cc:648] HybridClock initialized: now 1790363092711284 us; error 31 us; skew 500 ppm
Sep 25 19:04:52 dist-test-slave-rn98 krb5kdc[1857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:52.713825 2114 init.cc:377] Logged in from keytab as kudu/127.0.196.194@KRBTEST.COM (short username kudu)
I20260925 19:04:52.714314 2114 webserver.cc:460] Webserver started at http://127.0.196.194:38575/ using document root <none> and password file <none>
I20260925 19:04:52.714498 2114 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:52.714540 2114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:52.714694 2114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:52.715734 2114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-1/data/instance:
uuid: "82b4db6ddb0b412992bedd2ccde7d695"
format_stamp: "Formatted at 2026-09-25 19:04:52 on dist-test-slave-rn98"
server_key: "2b14ab371ec0378c8e0c7c3135f45ad8"
server_key_iv: "409b6d0dcc64d6ad4fda9df28c245ee8"
server_key_version: "encryptionkey@0"
I20260925 19:04:52.716051 2114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-1/wal/instance:
uuid: "82b4db6ddb0b412992bedd2ccde7d695"
format_stamp: "Formatted at 2026-09-25 19:04:52 on dist-test-slave-rn98"
server_key: "2b14ab371ec0378c8e0c7c3135f45ad8"
server_key_iv: "409b6d0dcc64d6ad4fda9df28c245ee8"
server_key_version: "encryptionkey@0"
I20260925 19:04:52.717367 2114 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 19:04:52.718104 2129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:52.718317 2114 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:04:52.718394 2114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "82b4db6ddb0b412992bedd2ccde7d695"
format_stamp: "Formatted at 2026-09-25 19:04:52 on dist-test-slave-rn98"
server_key: "2b14ab371ec0378c8e0c7c3135f45ad8"
server_key_iv: "409b6d0dcc64d6ad4fda9df28c245ee8"
server_key_version: "encryptionkey@0"
I20260925 19:04:52.718459 2114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:52.750406 2114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:52.752977 2114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:52.753165 2114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:52.753691 2114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:52.753736 2114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:52.753762 2114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:52.753799 2114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:52.761157 2114 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:45535
I20260925 19:04:52.761201 2244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:45535 every 8 connection(s)
I20260925 19:04:52.761536 2114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:04:52 dist-test-slave-rn98 krb5kdc[1857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:04:52.763312 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 2114
I20260925 19:04:52.763391 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-1/wal/instance
I20260925 19:04:52.763576 787 external_mini_cluster.cc:1428] Setting key 013e811d34ea1da6a426561b1fde70f2
I20260925 19:04:52.765560 1963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:52.754277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:56419 (local address 127.0.196.254:44651)
0925 19:04:52.754335 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:52.754356 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:52.754574 (+ 218us) server_negotiation.cc:407] Connection header received
0925 19:04:52.755171 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:52.755173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:52.755186 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:52.755256 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:52.756293 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.757110 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.758936 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.759274 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.759342 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:52.762219 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:52.762220 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:52.762221 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:52.762229 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:52.763847 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:52.764468 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:52.764470 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:52.764470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:52.764534 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:52.764797 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:52.764799 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:52.764799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:52.764930 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:52.765034 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:52.765362 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 19:04:52.765469 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 19:04:52.765763 2138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:52.754275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:44651 (local address 127.0.196.194:56419)
0925 19:04:52.754450 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:52.754487 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:52.755052 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:52.755383 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:52.755388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:52.755817 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:52.756169 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:52.756177 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.757217 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.757218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:52.758837 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:52.758840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.759375 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.759376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:52.759547 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:52.760259 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:52.760271 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:52.762111 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:52.764133 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:52.764139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:52.764141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:52.764374 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:52.764606 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:52.764607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:52.764608 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:52.764728 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:52.765026 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:52.765033 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:52.765213 (+ 180us) client_negotiation.cc:769] Sending connection context
0925 19:04:52.765354 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 19:04:52.765527 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
I20260925 19:04:52.766294 2245 heartbeater.cc:347] Connected to a master server at 127.0.196.254:44651
I20260925 19:04:52.766390 2245 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:52.766592 2245 heartbeater.cc:511] Master 127.0.196.254:44651 requested a full tablet report, sending...
I20260925 19:04:52.766993 1901 ts_manager.cc:194] Registered new tserver with Master: 82b4db6ddb0b412992bedd2ccde7d695 (127.0.196.194:45535)
I20260925 19:04:52.767545 1901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.194@KRBTEST.COM'} at 127.0.196.194:56419
WARNING: no policy specified for kudu/127.0.196.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.195@KRBTEST.COM" created.
I20260925 19:04:52.776409 2252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:52.767833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:44651 (local address 127.0.196.194:37001)
0925 19:04:52.770741 (+ 2908us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:52.770752 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:52.770832 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:52.770966 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:52.770967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:52.771150 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:52.771210 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:52.771212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.771872 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.771873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:52.772406 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:52.772407 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.772789 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.772790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:52.772858 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:52.773404 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:52.773409 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:52.773713 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:52.775229 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:52.775230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:52.775231 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:52.775438 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:52.775643 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:52.775644 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:52.775645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:52.775711 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:52.776050 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:52.776051 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:52.776115 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 19:04:52.776225 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 19:04:52.776329 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2878,"thread_start_us":80,"threads_started":1}
I20260925 19:04:52.776472 1963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:52.767890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:37001 (local address 127.0.196.254:44651)
0925 19:04:52.767929 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:52.767931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:52.770817 (+ 2886us) server_negotiation.cc:407] Connection header received
0925 19:04:52.770893 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:52.770894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:52.770905 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:52.770962 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:52.771287 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.771772 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.772493 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.772701 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.772788 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:52.773810 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:52.773812 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:52.773812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:52.773820 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:52.775127 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:52.775514 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:52.775515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:52.775516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:52.775577 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:52.775775 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:52.775776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:52.775776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:52.775939 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:52.776032 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:52.776266 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 19:04:52.776355 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
I20260925 19:04:52.791548 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:0
--local_ip_for_outbound_sockets=127.0.196.195
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:44651
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:52.882263 2254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:52.882488 2254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:52.882514 2254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:52.883872 2254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:52.883935 2254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:52.883958 2254 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 19:04:52.884012 2254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:04:52.885601 2254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.196.254:44651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:52.885871 2254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:52.886118 2254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:04:52.891669 2254 server_base.cc:1034] running on GCE node
W20260925 19:04:52.891616 2259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:52.891633 2262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:52.891633 2260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:52.892145 2254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:52.892470 2254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:52.893663 2254 hybrid_clock.cc:648] HybridClock initialized: now 1790363092893626 us; error 53 us; skew 500 ppm
Sep 25 19:04:52 dist-test-slave-rn98 krb5kdc[1857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:52.896680 2254 init.cc:377] Logged in from keytab as kudu/127.0.196.195@KRBTEST.COM (short username kudu)
I20260925 19:04:52.897370 2254 webserver.cc:460] Webserver started at http://127.0.196.195:34689/ using document root <none> and password file <none>
I20260925 19:04:52.897629 2254 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:52.897677 2254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:52.897821 2254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:52.899078 2254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-2/data/instance:
uuid: "11546da847ba4de3a92d36e61708a7c4"
format_stamp: "Formatted at 2026-09-25 19:04:52 on dist-test-slave-rn98"
server_key: "ad9b7d63ba640aa3ee47c0013d289f88"
server_key_iv: "c626d0715c9fc261d919db83cf967626"
server_key_version: "encryptionkey@0"
I20260925 19:04:52.899425 2254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-2/wal/instance:
uuid: "11546da847ba4de3a92d36e61708a7c4"
format_stamp: "Formatted at 2026-09-25 19:04:52 on dist-test-slave-rn98"
server_key: "ad9b7d63ba640aa3ee47c0013d289f88"
server_key_iv: "c626d0715c9fc261d919db83cf967626"
server_key_version: "encryptionkey@0"
I20260925 19:04:52.900877 2254 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:04:52.901715 2269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:52.902094 2254 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:04:52.902170 2254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "11546da847ba4de3a92d36e61708a7c4"
format_stamp: "Formatted at 2026-09-25 19:04:52 on dist-test-slave-rn98"
server_key: "ad9b7d63ba640aa3ee47c0013d289f88"
server_key_iv: "c626d0715c9fc261d919db83cf967626"
server_key_version: "encryptionkey@0"
I20260925 19:04:52.902262 2254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:52.927902 2254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:52.930024 2254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:52.930177 2254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:52.930941 2254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:52.931007 2254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:52.931041 2254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:52.931066 2254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:52.938185 2254 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:39297
I20260925 19:04:52.938246 2384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:39297 every 8 connection(s)
Sep 25 19:04:52 dist-test-slave-rn98 krb5kdc[1857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:04:52.938576 2254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
I20260925 19:04:52.942281 1963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:52.931370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:39201 (local address 127.0.196.254:44651)
0925 19:04:52.931440 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:52.931443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:52.931628 (+ 185us) server_negotiation.cc:407] Connection header received
0925 19:04:52.932198 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:52.932207 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:52.932220 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:52.932273 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:52.933176 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.933980 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.935793 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.936140 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.936230 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:52.939250 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:52.939252 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:52.939252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:52.939259 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:52.940852 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:52.941282 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:52.941283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:52.941284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:52.941345 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:52.941607 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:52.941607 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:52.941608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:52.941756 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:52.941813 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:52.942089 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 19:04:52.942169 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 19:04:52.942334 2278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:52.931295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:44651 (local address 127.0.196.195:39201)
0925 19:04:52.931489 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:52.931539 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:52.932086 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:52.932368 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:52.932372 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:52.932784 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:52.933089 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:52.933096 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.934098 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.934099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:52.935662 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:52.935665 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.936243 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.936244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:52.936431 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:52.937172 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:52.937185 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:52.939161 (+ 1976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:52.940931 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:52.940937 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:52.940939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:52.941170 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:52.941417 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:52.941417 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:52.941417 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:52.941518 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:52.941817 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:52.941821 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:52.941949 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 19:04:52.942039 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 19:04:52.942166 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":78,"threads_started":1}
I20260925 19:04:52.942845 2385 heartbeater.cc:347] Connected to a master server at 127.0.196.254:44651
I20260925 19:04:52.942961 2385 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:52.943143 2385 heartbeater.cc:511] Master 127.0.196.254:44651 requested a full tablet report, sending...
I20260925 19:04:52.943588 1902 ts_manager.cc:194] Registered new tserver with Master: 11546da847ba4de3a92d36e61708a7c4 (127.0.196.195:39297)
I20260925 19:04:52.944070 1902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.195@KRBTEST.COM'} at 127.0.196.195:39201
I20260925 19:04:52.948141 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 2254
I20260925 19:04:52.948240 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-2/wal/instance
I20260925 19:04:52.948413 787 external_mini_cluster.cc:1428] Setting key 87b15749904e2089c46dea2b1702b5a2
I20260925 19:04:52.949331 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:04:52.950331 1963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:52.944002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:39919 (local address 127.0.196.254:44651)
0925 19:04:52.944044 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:52.944055 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:52.944121 (+ 66us) server_negotiation.cc:407] Connection header received
0925 19:04:52.944182 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:52.944183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:52.944200 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:52.944258 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:52.944665 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.945310 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.945985 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.946171 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.946228 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:52.947256 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:52.947257 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:52.947258 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:52.947266 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:52.949061 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:52.949514 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:52.949515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:52.949515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:52.949569 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:52.949805 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:52.949806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:52.949806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:52.949928 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:52.949990 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:52.950175 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 19:04:52.950244 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:04:52.950417 2391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:52.943924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:44651 (local address 127.0.196.195:39919)
0925 19:04:52.944056 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:52.944068 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:52.944132 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:52.944269 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:52.944270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:52.944474 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:52.944550 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:52.944552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.945408 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.945409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:52.945911 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:52.945912 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.946237 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.946238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:52.946290 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:52.946856 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:52.946860 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:52.947161 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:52.949145 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:52.949146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:52.949147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:52.949400 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:52.949633 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:52.949634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:52.949635 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:52.949717 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:52.950000 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:52.950001 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:52.950061 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 19:04:52.950136 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 19:04:52.950265 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
Sep 25 19:04:52 dist-test-slave-rn98 krb5kdc[1857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363092, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 19:04:52 dist-test-slave-rn98 krb5kdc[1857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363092, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:04:52.983057 1963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:52.975324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41992 (local address 127.0.196.254:44651)
0925 19:04:52.975393 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:52.975396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:52.975409 (+ 13us) server_negotiation.cc:407] Connection header received
0925 19:04:52.975444 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:52.975445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:52.975458 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:52.975545 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:52.976211 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.976878 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.977798 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.978013 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.978092 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:52.980532 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:52.980533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:52.980533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:52.980555 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:52.981919 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:52.982261 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:52.982262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:52.982262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:52.982312 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:52.982476 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:52.982477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:52.982477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:52.982578 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:52.982707 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:52.982914 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 19:04:52.982988 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 19:04:52.984899 1902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:04:52.986683 1902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:04:52.993216 2402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:52.989002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:39297 (local address 127.0.0.1:54780)
0925 19:04:52.989211 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:52.989257 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:52.989334 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:52.989862 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:52.989862 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:52.989864 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:52.989913 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:52.989914 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.991206 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.991207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:52.992103 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:52.992106 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.992891 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.992892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:52.992987 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:52.993005 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 19:04:52.993079 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 19:04:52.993129 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":50,"thread_start_us":47,"threads_started":1}
I20260925 19:04:52.993250 2404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:52.989095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54780 (local address 127.0.196.195:39297)
0925 19:04:52.989722 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:52.989726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:52.989735 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:04:52.989772 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:52.989775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:52.989790 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:52.989868 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:52.989999 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.991076 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.992217 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.992813 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.992885 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:52.992965 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:52.993105 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 19:04:52.993146 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":97,"threads_started":1}
I20260925 19:04:52.993738 2405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:52.989182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:36747 (local address 127.0.0.1:39954)
0925 19:04:52.989963 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:52.989974 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:52.990023 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:52.990225 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:52.990226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:52.990228 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:52.990271 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:52.990273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.991225 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.991226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:52.992430 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:52.992433 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.993513 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.993514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:52.993556 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:52.993559 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:04:52.993627 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 19:04:52.993677 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":42,"thread_start_us":73,"threads_started":1}
I20260925 19:04:52.994000 2403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:52.989002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:45535 (local address 127.0.0.1:35424)
0925 19:04:52.989276 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:52.989287 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:52.989344 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:52.989609 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:52.989611 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:52.989614 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:52.989702 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:52.989704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.991106 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.991107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:52.992696 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:52.992698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.993718 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.993719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:52.993827 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:52.993831 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:04:52.993891 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 19:04:52.993934 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":59,"thread_start_us":85,"threads_started":1}
I20260925 19:04:52.994103 2407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:52.989397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39954 (local address 127.0.196.193:36747)
0925 19:04:52.989545 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:52.989549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:52.990077 (+ 528us) server_negotiation.cc:407] Connection header received
0925 19:04:52.990124 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:52.990125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:52.990137 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:52.990217 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:52.990325 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.991136 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.992519 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.993345 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.993411 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:52.993480 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:52.993919 (+ 439us) server_negotiation.cc:299] Negotiation successful
0925 19:04:52.993966 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260925 19:04:52.994294 2406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:52.989191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35424 (local address 127.0.196.194:45535)
0925 19:04:52.989424 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:52.989430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:52.989442 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:04:52.989487 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:52.989489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:52.989505 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:52.989598 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:52.989783 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.991018 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.992773 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:52.993536 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:52.993610 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:52.993683 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:52.994148 (+ 465us) server_negotiation.cc:299] Negotiation successful
0925 19:04:52.994189 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":123,"threads_started":1}
I20260925 19:04:52.994221 2319 tablet_service.cc:1467] Processing CreateTablet for tablet 414707a212664effb9775a3fe9da5afe (DEFAULT_TABLE table=test-table [id=7bce54231127408896b022499604616b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:52.994702 2319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 414707a212664effb9775a3fe9da5afe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:52.994793 2039 tablet_service.cc:1467] Processing CreateTablet for tablet 414707a212664effb9775a3fe9da5afe (DEFAULT_TABLE table=test-table [id=7bce54231127408896b022499604616b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:52.994922 2179 tablet_service.cc:1467] Processing CreateTablet for tablet 414707a212664effb9775a3fe9da5afe (DEFAULT_TABLE table=test-table [id=7bce54231127408896b022499604616b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:52.995182 2039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 414707a212664effb9775a3fe9da5afe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:52.995258 2179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 414707a212664effb9775a3fe9da5afe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:52.998407 2408 tablet_bootstrap.cc:492] T 414707a212664effb9775a3fe9da5afe P 3db90292012d4a61904ea62ffb27dc8f: Bootstrap starting.
I20260925 19:04:52.998723 2409 tablet_bootstrap.cc:492] T 414707a212664effb9775a3fe9da5afe P 11546da847ba4de3a92d36e61708a7c4: Bootstrap starting.
I20260925 19:04:52.999269 2408 tablet_bootstrap.cc:654] T 414707a212664effb9775a3fe9da5afe P 3db90292012d4a61904ea62ffb27dc8f: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:52.999315 2409 tablet_bootstrap.cc:654] T 414707a212664effb9775a3fe9da5afe P 11546da847ba4de3a92d36e61708a7c4: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:52.999356 2410 tablet_bootstrap.cc:492] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695: Bootstrap starting.
I20260925 19:04:52.999652 2409 log.cc:824] T 414707a212664effb9775a3fe9da5afe P 11546da847ba4de3a92d36e61708a7c4: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:52.999747 2408 log.cc:824] T 414707a212664effb9775a3fe9da5afe P 3db90292012d4a61904ea62ffb27dc8f: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:52.999985 2410 tablet_bootstrap.cc:654] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:53.000310 2410 log.cc:824] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:53.000440 2409 tablet_bootstrap.cc:492] T 414707a212664effb9775a3fe9da5afe P 11546da847ba4de3a92d36e61708a7c4: No bootstrap required, opened a new log
I20260925 19:04:53.000517 2409 ts_tablet_manager.cc:1397] T 414707a212664effb9775a3fe9da5afe P 11546da847ba4de3a92d36e61708a7c4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:04:53.000588 2408 tablet_bootstrap.cc:492] T 414707a212664effb9775a3fe9da5afe P 3db90292012d4a61904ea62ffb27dc8f: No bootstrap required, opened a new log
I20260925 19:04:53.000658 2408 ts_tablet_manager.cc:1397] T 414707a212664effb9775a3fe9da5afe P 3db90292012d4a61904ea62ffb27dc8f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:04:53.000993 2410 tablet_bootstrap.cc:492] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695: No bootstrap required, opened a new log
I20260925 19:04:53.001063 2410 ts_tablet_manager.cc:1397] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:04:53.002101 2408 raft_consensus.cc:348] T 414707a212664effb9775a3fe9da5afe P 3db90292012d4a61904ea62ffb27dc8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } }
I20260925 19:04:53.002269 2408 raft_consensus.cc:374] T 414707a212664effb9775a3fe9da5afe P 3db90292012d4a61904ea62ffb27dc8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:53.002313 2408 raft_consensus.cc:729] T 414707a212664effb9775a3fe9da5afe P 3db90292012d4a61904ea62ffb27dc8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db90292012d4a61904ea62ffb27dc8f, State: Initialized, Role: FOLLOWER
I20260925 19:04:53.002449 2408 consensus_queue.cc:260] T 414707a212664effb9775a3fe9da5afe P 3db90292012d4a61904ea62ffb27dc8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } }
I20260925 19:04:53.002578 2409 raft_consensus.cc:348] T 414707a212664effb9775a3fe9da5afe P 11546da847ba4de3a92d36e61708a7c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } }
I20260925 19:04:53.002800 2409 raft_consensus.cc:374] T 414707a212664effb9775a3fe9da5afe P 11546da847ba4de3a92d36e61708a7c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:53.002776 2408 ts_tablet_manager.cc:1428] T 414707a212664effb9775a3fe9da5afe P 3db90292012d4a61904ea62ffb27dc8f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:04:53.002844 2409 raft_consensus.cc:729] T 414707a212664effb9775a3fe9da5afe P 11546da847ba4de3a92d36e61708a7c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11546da847ba4de3a92d36e61708a7c4, State: Initialized, Role: FOLLOWER
I20260925 19:04:53.002928 2105 heartbeater.cc:503] Master 127.0.196.254:44651 was elected leader, sending a full tablet report...
I20260925 19:04:53.002961 2409 consensus_queue.cc:260] T 414707a212664effb9775a3fe9da5afe P 11546da847ba4de3a92d36e61708a7c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } }
I20260925 19:04:53.003051 2410 raft_consensus.cc:348] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } }
I20260925 19:04:53.003204 2410 raft_consensus.cc:374] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:53.003221 2409 ts_tablet_manager.cc:1428] T 414707a212664effb9775a3fe9da5afe P 11546da847ba4de3a92d36e61708a7c4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:04:53.003245 2410 raft_consensus.cc:729] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82b4db6ddb0b412992bedd2ccde7d695, State: Initialized, Role: FOLLOWER
I20260925 19:04:53.003329 2385 heartbeater.cc:503] Master 127.0.196.254:44651 was elected leader, sending a full tablet report...
I20260925 19:04:53.003362 2410 consensus_queue.cc:260] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } }
I20260925 19:04:53.003609 2410 ts_tablet_manager.cc:1428] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:04:53.003746 2245 heartbeater.cc:503] Master 127.0.196.254:44651 was elected leader, sending a full tablet report...
W20260925 19:04:53.012106 2246 tablet.cc:2369] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:04:53.018052 2416 raft_consensus.cc:482] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:04:53.018170 2416 raft_consensus.cc:504] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } }
I20260925 19:04:53.018698 2416 leader_election.cc:288] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11546da847ba4de3a92d36e61708a7c4 (127.0.196.195:39297), 3db90292012d4a61904ea62ffb27dc8f (127.0.196.193:36747)
I20260925 19:04:53.022742 2407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:53.018952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:53395 (local address 127.0.196.193:36747)
0925 19:04:53.019039 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:53.019042 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:53.019257 (+ 215us) server_negotiation.cc:407] Connection header received
0925 19:04:53.019315 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:53.019316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:53.019323 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:53.019370 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:53.019520 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.020205 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.021730 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.022335 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.022412 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:53.022421 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:53.022595 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 19:04:53.022674 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 19:04:53.022797 2417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:53.018885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:36747 (local address 127.0.196.194:53395)
0925 19:04:53.019159 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:53.019168 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:53.019254 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:53.019389 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:53.019390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:53.019391 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:53.019460 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:53.019461 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.020294 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.020295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:53.021634 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:53.021637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.022418 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.022419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:53.022518 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:53.022523 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:04:53.022585 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 19:04:53.022663 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":98,"thread_start_us":136,"threads_started":1}
I20260925 19:04:53.023132 2138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:53.018876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:39297 (local address 127.0.196.194:33997)
0925 19:04:53.018963 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:53.018978 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:53.019060 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:53.019233 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:53.019234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:53.019236 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:53.019318 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:53.019320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.020294 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.020295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:53.021633 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:53.021636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.022418 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.022419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:53.022946 (+ 527us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:53.022955 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 19:04:53.023031 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 19:04:53.023070 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":65}
I20260925 19:04:53.023144 2404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:53.018942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:33997 (local address 127.0.196.195:39297)
0925 19:04:53.018991 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:53.018993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:53.019054 (+ 61us) server_negotiation.cc:407] Connection header received
0925 19:04:53.019150 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:53.019152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:53.019160 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:53.019209 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:53.019422 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.020195 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.021731 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.022325 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.022412 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:53.022421 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:53.023038 (+ 617us) server_negotiation.cc:299] Negotiation successful
0925 19:04:53.023082 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 19:04:53.023160 2059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414707a212664effb9775a3fe9da5afe" candidate_uuid: "82b4db6ddb0b412992bedd2ccde7d695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3db90292012d4a61904ea62ffb27dc8f" is_pre_election: true
I20260925 19:04:53.023335 2059 raft_consensus.cc:2454] T 414707a212664effb9775a3fe9da5afe P 3db90292012d4a61904ea62ffb27dc8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82b4db6ddb0b412992bedd2ccde7d695 in term 0.
I20260925 19:04:53.023465 2339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414707a212664effb9775a3fe9da5afe" candidate_uuid: "82b4db6ddb0b412992bedd2ccde7d695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11546da847ba4de3a92d36e61708a7c4" is_pre_election: true
I20260925 19:04:53.023557 2130 leader_election.cc:302] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3db90292012d4a61904ea62ffb27dc8f, 82b4db6ddb0b412992bedd2ccde7d695; no voters:
I20260925 19:04:53.023618 2339 raft_consensus.cc:2454] T 414707a212664effb9775a3fe9da5afe P 11546da847ba4de3a92d36e61708a7c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82b4db6ddb0b412992bedd2ccde7d695 in term 0.
I20260925 19:04:53.023737 2416 raft_consensus.cc:2781] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:04:53.023809 2416 raft_consensus.cc:482] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:04:53.023839 2416 raft_consensus.cc:3037] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:53.024677 2416 raft_consensus.cc:504] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } }
I20260925 19:04:53.024833 2416 leader_election.cc:288] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695 [CANDIDATE]: Term 1 election: Requested vote from peers 11546da847ba4de3a92d36e61708a7c4 (127.0.196.195:39297), 3db90292012d4a61904ea62ffb27dc8f (127.0.196.193:36747)
I20260925 19:04:53.024983 2339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414707a212664effb9775a3fe9da5afe" candidate_uuid: "82b4db6ddb0b412992bedd2ccde7d695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11546da847ba4de3a92d36e61708a7c4"
I20260925 19:04:53.025064 2339 raft_consensus.cc:3037] T 414707a212664effb9775a3fe9da5afe P 11546da847ba4de3a92d36e61708a7c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:53.025755 2339 raft_consensus.cc:2454] T 414707a212664effb9775a3fe9da5afe P 11546da847ba4de3a92d36e61708a7c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82b4db6ddb0b412992bedd2ccde7d695 in term 1.
I20260925 19:04:53.025874 2059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414707a212664effb9775a3fe9da5afe" candidate_uuid: "82b4db6ddb0b412992bedd2ccde7d695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3db90292012d4a61904ea62ffb27dc8f"
I20260925 19:04:53.025925 2132 leader_election.cc:302] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11546da847ba4de3a92d36e61708a7c4, 82b4db6ddb0b412992bedd2ccde7d695; no voters:
I20260925 19:04:53.025959 2059 raft_consensus.cc:3037] T 414707a212664effb9775a3fe9da5afe P 3db90292012d4a61904ea62ffb27dc8f [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:53.026072 2416 raft_consensus.cc:2781] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:53.026255 2416 raft_consensus.cc:686] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695 [term 1 LEADER]: Becoming Leader. State: Replica: 82b4db6ddb0b412992bedd2ccde7d695, State: Running, Role: LEADER
I20260925 19:04:53.026357 2416 consensus_queue.cc:237] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } }
I20260925 19:04:53.026923 2059 raft_consensus.cc:2454] T 414707a212664effb9775a3fe9da5afe P 3db90292012d4a61904ea62ffb27dc8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82b4db6ddb0b412992bedd2ccde7d695 in term 1.
I20260925 19:04:53.027257 1902 catalog_manager.cc:5495] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82b4db6ddb0b412992bedd2ccde7d695 (127.0.196.194). New cstate: current_term: 1 leader_uuid: "82b4db6ddb0b412992bedd2ccde7d695" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } health_report { overall_health: UNKNOWN } } }
I20260925 19:04:53.039671 2406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:53.037261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35430 (local address 127.0.196.194:45535)
0925 19:04:53.037307 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:53.037309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:53.037325 (+ 16us) server_negotiation.cc:407] Connection header received
0925 19:04:53.037358 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:53.037359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:53.037375 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:53.037439 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:53.037585 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.038084 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.038938 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.039158 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.039220 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:53.039403 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:53.039496 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:53.039556 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:53.039598 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 19:04:53.039615 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:04:53.041975 2059 raft_consensus.cc:1261] T 414707a212664effb9775a3fe9da5afe P 3db90292012d4a61904ea62ffb27dc8f [term 1 FOLLOWER]: Refusing update from remote peer 82b4db6ddb0b412992bedd2ccde7d695: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:53.041996 2339 raft_consensus.cc:1261] T 414707a212664effb9775a3fe9da5afe P 11546da847ba4de3a92d36e61708a7c4 [term 1 FOLLOWER]: Refusing update from remote peer 82b4db6ddb0b412992bedd2ccde7d695: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:53.042412 2418 consensus_queue.cc:1035] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:53.042573 2416 consensus_queue.cc:1035] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:53.048393 1902 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:04:53.048478 1902 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:04:53.049223 1902 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4fede94c0b9b47c3a3b92e5d8df187cb: Sending DeleteTablet for 3 replicas of tablet 414707a212664effb9775a3fe9da5afe
I20260925 19:04:53.049552 2179 tablet_service.cc:1514] Processing DeleteTablet for tablet 414707a212664effb9775a3fe9da5afe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:53 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:35424
I20260925 19:04:53.049595 2039 tablet_service.cc:1514] Processing DeleteTablet for tablet 414707a212664effb9775a3fe9da5afe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:53 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:39954
I20260925 19:04:53.049810 2431 tablet_replica.cc:330] T 414707a212664effb9775a3fe9da5afe P 3db90292012d4a61904ea62ffb27dc8f: stopping tablet replica
I20260925 19:04:53.049805 2319 tablet_service.cc:1514] Processing DeleteTablet for tablet 414707a212664effb9775a3fe9da5afe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:53 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:54780
I20260925 19:04:53.049924 2431 raft_consensus.cc:2229] T 414707a212664effb9775a3fe9da5afe P 3db90292012d4a61904ea62ffb27dc8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:04:53.050041 2431 raft_consensus.cc:2258] T 414707a212664effb9775a3fe9da5afe P 3db90292012d4a61904ea62ffb27dc8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:53.050046 2433 tablet_replica.cc:330] T 414707a212664effb9775a3fe9da5afe P 11546da847ba4de3a92d36e61708a7c4: stopping tablet replica
I20260925 19:04:53.050215 2432 tablet_replica.cc:330] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695: stopping tablet replica
I20260925 19:04:53.050226 2433 raft_consensus.cc:2229] T 414707a212664effb9775a3fe9da5afe P 11546da847ba4de3a92d36e61708a7c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:04:53.050308 2432 raft_consensus.cc:2229] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:04:53.050298 1902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 19:04:53.050414 2432 raft_consensus.cc:2258] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 19:04:53.050449 1902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:04:53.050673 2433 raft_consensus.cc:2258] T 414707a212664effb9775a3fe9da5afe P 11546da847ba4de3a92d36e61708a7c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:53.051059 2431 ts_tablet_manager.cc:1905] T 414707a212664effb9775a3fe9da5afe P 3db90292012d4a61904ea62ffb27dc8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:53.051261 2433 ts_tablet_manager.cc:1905] T 414707a212664effb9775a3fe9da5afe P 11546da847ba4de3a92d36e61708a7c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:53.051818 2039 tablet_service.cc:1467] Processing CreateTablet for tablet 7620dec3ee104dfd8ffe7fe2652de9f7 (DEFAULT_TABLE table=test-table [id=09f826bc48bc4f78ae8694f0417a86df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:53.051843 2179 tablet_service.cc:1467] Processing CreateTablet for tablet 7620dec3ee104dfd8ffe7fe2652de9f7 (DEFAULT_TABLE table=test-table [id=09f826bc48bc4f78ae8694f0417a86df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:53.051935 2039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7620dec3ee104dfd8ffe7fe2652de9f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:53.051955 2179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7620dec3ee104dfd8ffe7fe2652de9f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:53.053176 2408 tablet_bootstrap.cc:492] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 3db90292012d4a61904ea62ffb27dc8f: Bootstrap starting.
I20260925 19:04:53.053313 2410 tablet_bootstrap.cc:492] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 82b4db6ddb0b412992bedd2ccde7d695: Bootstrap starting.
I20260925 19:04:53.053597 2408 tablet_bootstrap.cc:654] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 3db90292012d4a61904ea62ffb27dc8f: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:53.053692 2433 ts_tablet_manager.cc:1918] T 414707a212664effb9775a3fe9da5afe P 11546da847ba4de3a92d36e61708a7c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:04:53.053741 2410 tablet_bootstrap.cc:654] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 82b4db6ddb0b412992bedd2ccde7d695: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:53.053867 2433 log.cc:1196] T 414707a212664effb9775a3fe9da5afe P 11546da847ba4de3a92d36e61708a7c4: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-2/wal/wals/414707a212664effb9775a3fe9da5afe
I20260925 19:04:53.053936 2431 ts_tablet_manager.cc:1918] T 414707a212664effb9775a3fe9da5afe P 3db90292012d4a61904ea62ffb27dc8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:04:53.054010 2431 log.cc:1196] T 414707a212664effb9775a3fe9da5afe P 3db90292012d4a61904ea62ffb27dc8f: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-0/wal/wals/414707a212664effb9775a3fe9da5afe
I20260925 19:04:53.054255 2431 ts_tablet_manager.cc:1939] T 414707a212664effb9775a3fe9da5afe P 3db90292012d4a61904ea62ffb27dc8f: Deleting consensus metadata
I20260925 19:04:53.054673 1886 catalog_manager.cc:4841] TS 3db90292012d4a61904ea62ffb27dc8f (127.0.196.193:36747): tablet 414707a212664effb9775a3fe9da5afe (table test-table [id=7bce54231127408896b022499604616b]) successfully deleted
I20260925 19:04:53.054767 2319 tablet_service.cc:1467] Processing CreateTablet for tablet 7620dec3ee104dfd8ffe7fe2652de9f7 (DEFAULT_TABLE table=test-table [id=09f826bc48bc4f78ae8694f0417a86df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:53.054894 2319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7620dec3ee104dfd8ffe7fe2652de9f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:53.055279 2408 tablet_bootstrap.cc:492] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 3db90292012d4a61904ea62ffb27dc8f: No bootstrap required, opened a new log
I20260925 19:04:53.055320 2408 ts_tablet_manager.cc:1397] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 3db90292012d4a61904ea62ffb27dc8f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 19:04:53.055413 2432 ts_tablet_manager.cc:1905] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:53.055454 2408 raft_consensus.cc:348] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 3db90292012d4a61904ea62ffb27dc8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } }
I20260925 19:04:53.055518 2408 raft_consensus.cc:374] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 3db90292012d4a61904ea62ffb27dc8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:53.055539 2408 raft_consensus.cc:729] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 3db90292012d4a61904ea62ffb27dc8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db90292012d4a61904ea62ffb27dc8f, State: Initialized, Role: FOLLOWER
I20260925 19:04:53.055585 2408 consensus_queue.cc:260] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 3db90292012d4a61904ea62ffb27dc8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } }
I20260925 19:04:53.055665 2408 ts_tablet_manager.cc:1428] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 3db90292012d4a61904ea62ffb27dc8f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:53.056239 2409 tablet_bootstrap.cc:492] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4: Bootstrap starting.
I20260925 19:04:53.056630 2409 tablet_bootstrap.cc:654] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:53.057020 2432 ts_tablet_manager.cc:1918] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:04:53.057085 2432 log.cc:1196] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-1/wal/wals/414707a212664effb9775a3fe9da5afe
I20260925 19:04:53.054809 2433 ts_tablet_manager.cc:1939] T 414707a212664effb9775a3fe9da5afe P 11546da847ba4de3a92d36e61708a7c4: Deleting consensus metadata
I20260925 19:04:53.057294 2409 tablet_bootstrap.cc:492] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4: No bootstrap required, opened a new log
I20260925 19:04:53.057359 2432 ts_tablet_manager.cc:1939] T 414707a212664effb9775a3fe9da5afe P 82b4db6ddb0b412992bedd2ccde7d695: Deleting consensus metadata
I20260925 19:04:53.057400 2409 ts_tablet_manager.cc:1397] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:04:53.057555 2409 raft_consensus.cc:348] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } }
I20260925 19:04:53.057616 2409 raft_consensus.cc:374] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:53.057638 2409 raft_consensus.cc:729] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11546da847ba4de3a92d36e61708a7c4, State: Initialized, Role: FOLLOWER
I20260925 19:04:53.057684 2409 consensus_queue.cc:260] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } }
I20260925 19:04:53.057770 2409 ts_tablet_manager.cc:1428] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:53.057816 1886 catalog_manager.cc:4841] TS 82b4db6ddb0b412992bedd2ccde7d695 (127.0.196.194:45535): tablet 414707a212664effb9775a3fe9da5afe (table test-table [id=7bce54231127408896b022499604616b]) successfully deleted
I20260925 19:04:53.058432 1888 catalog_manager.cc:4841] TS 11546da847ba4de3a92d36e61708a7c4 (127.0.196.195:39297): tablet 414707a212664effb9775a3fe9da5afe (table test-table [id=7bce54231127408896b022499604616b]) successfully deleted
I20260925 19:04:53.058779 2410 tablet_bootstrap.cc:492] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 82b4db6ddb0b412992bedd2ccde7d695: No bootstrap required, opened a new log
I20260925 19:04:53.058827 2410 ts_tablet_manager.cc:1397] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 82b4db6ddb0b412992bedd2ccde7d695: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260925 19:04:53.059010 2410 raft_consensus.cc:348] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 82b4db6ddb0b412992bedd2ccde7d695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } }
I20260925 19:04:53.059093 2410 raft_consensus.cc:374] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 82b4db6ddb0b412992bedd2ccde7d695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:53.059119 2410 raft_consensus.cc:729] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 82b4db6ddb0b412992bedd2ccde7d695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82b4db6ddb0b412992bedd2ccde7d695, State: Initialized, Role: FOLLOWER
I20260925 19:04:53.059165 2410 consensus_queue.cc:260] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 82b4db6ddb0b412992bedd2ccde7d695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } }
I20260925 19:04:53.059257 2410 ts_tablet_manager.cc:1428] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 82b4db6ddb0b412992bedd2ccde7d695: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260925 19:04:53.088920 2106 tablet.cc:2369] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 3db90292012d4a61904ea62ffb27dc8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:04:53.163307 2415 raft_consensus.cc:482] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:04:53.163426 2415 raft_consensus.cc:504] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } }
I20260925 19:04:53.163745 2415 leader_election.cc:288] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3db90292012d4a61904ea62ffb27dc8f (127.0.196.193:36747), 82b4db6ddb0b412992bedd2ccde7d695 (127.0.196.194:45535)
I20260925 19:04:53.167764 2278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:53.163970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:36747 (local address 127.0.196.195:39371)
0925 19:04:53.164057 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:53.164072 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:53.164148 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:53.164307 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:53.164308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:53.164311 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:53.164422 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:53.164439 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.165366 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.165367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:53.166721 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:53.166724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.167514 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.167515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:53.167587 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:53.167589 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:04:53.167646 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 19:04:53.167688 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":5}
I20260925 19:04:53.167763 2406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:53.164177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:60267 (local address 127.0.196.194:45535)
0925 19:04:53.164218 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:53.164219 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:53.164298 (+ 79us) server_negotiation.cc:407] Connection header received
0925 19:04:53.164383 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:53.164385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:53.164395 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:53.164447 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:53.164614 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.165399 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.166907 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.167374 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.167442 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:53.167452 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:53.167647 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 19:04:53.167686 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:04:53.167778 2407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:53.164036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:39371 (local address 127.0.196.193:36747)
0925 19:04:53.164082 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:53.164084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:53.164187 (+ 103us) server_negotiation.cc:407] Connection header received
0925 19:04:53.164223 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:53.164224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:53.164234 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:53.164289 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:53.164531 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.165267 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.166819 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.167438 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.167508 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:53.167516 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:53.167665 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 19:04:53.167703 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:04:53.168175 2437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:53.164060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:45535 (local address 127.0.196.195:60267)
0925 19:04:53.164243 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:53.164251 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:53.164320 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:53.164460 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:53.164461 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:53.164462 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:53.164542 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:53.164543 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.165464 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.165464 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:53.166721 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:53.166724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.167484 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.167485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:53.167555 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:53.167560 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:04:53.168068 (+ 508us) client_negotiation.cc:240] Negotiation successful
0925 19:04:53.168109 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":3,"thread_start_us":55,"threads_started":1}
I20260925 19:04:53.168140 2059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7620dec3ee104dfd8ffe7fe2652de9f7" candidate_uuid: "11546da847ba4de3a92d36e61708a7c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3db90292012d4a61904ea62ffb27dc8f" is_pre_election: true
I20260925 19:04:53.168221 2059 raft_consensus.cc:2454] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 3db90292012d4a61904ea62ffb27dc8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11546da847ba4de3a92d36e61708a7c4 in term 0.
I20260925 19:04:53.168459 2270 leader_election.cc:302] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11546da847ba4de3a92d36e61708a7c4, 3db90292012d4a61904ea62ffb27dc8f; no voters:
I20260925 19:04:53.168637 2415 raft_consensus.cc:2781] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:04:53.168701 2415 raft_consensus.cc:482] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:04:53.168735 2415 raft_consensus.cc:3037] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:53.168723 2199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7620dec3ee104dfd8ffe7fe2652de9f7" candidate_uuid: "11546da847ba4de3a92d36e61708a7c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82b4db6ddb0b412992bedd2ccde7d695" is_pre_election: true
I20260925 19:04:53.168859 2199 raft_consensus.cc:2454] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 82b4db6ddb0b412992bedd2ccde7d695 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11546da847ba4de3a92d36e61708a7c4 in term 0.
I20260925 19:04:53.169720 2415 raft_consensus.cc:504] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } }
I20260925 19:04:53.169853 2415 leader_election.cc:288] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4 [CANDIDATE]: Term 1 election: Requested vote from peers 3db90292012d4a61904ea62ffb27dc8f (127.0.196.193:36747), 82b4db6ddb0b412992bedd2ccde7d695 (127.0.196.194:45535)
I20260925 19:04:53.170074 2059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7620dec3ee104dfd8ffe7fe2652de9f7" candidate_uuid: "11546da847ba4de3a92d36e61708a7c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3db90292012d4a61904ea62ffb27dc8f"
I20260925 19:04:53.170075 2199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7620dec3ee104dfd8ffe7fe2652de9f7" candidate_uuid: "11546da847ba4de3a92d36e61708a7c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82b4db6ddb0b412992bedd2ccde7d695"
I20260925 19:04:53.170152 2059 raft_consensus.cc:3037] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 3db90292012d4a61904ea62ffb27dc8f [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:53.170167 2199 raft_consensus.cc:3037] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 82b4db6ddb0b412992bedd2ccde7d695 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:53.170912 2059 raft_consensus.cc:2454] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 3db90292012d4a61904ea62ffb27dc8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11546da847ba4de3a92d36e61708a7c4 in term 1.
I20260925 19:04:53.170928 2199 raft_consensus.cc:2454] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 82b4db6ddb0b412992bedd2ccde7d695 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11546da847ba4de3a92d36e61708a7c4 in term 1.
I20260925 19:04:53.171075 2270 leader_election.cc:302] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11546da847ba4de3a92d36e61708a7c4, 82b4db6ddb0b412992bedd2ccde7d695; no voters:
I20260925 19:04:53.171208 2415 raft_consensus.cc:2781] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:53.171339 2415 raft_consensus.cc:686] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4 [term 1 LEADER]: Becoming Leader. State: Replica: 11546da847ba4de3a92d36e61708a7c4, State: Running, Role: LEADER
I20260925 19:04:53.171416 2415 consensus_queue.cc:237] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } }
I20260925 19:04:53.172211 1902 catalog_manager.cc:5495] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11546da847ba4de3a92d36e61708a7c4 (127.0.196.195). New cstate: current_term: 1 leader_uuid: "11546da847ba4de3a92d36e61708a7c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } health_report { overall_health: HEALTHY } } }
W20260925 19:04:53.189211 2386 tablet.cc:2369] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 19:04:53 dist-test-slave-rn98 krb5kdc[1857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:04:53.225862 2445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:53.219325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:44651 (local address 127.0.0.1:41996)
0925 19:04:53.219463 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:53.219474 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:53.219554 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:53.219695 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:53.219695 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:53.219999 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:53.220045 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:53.220047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.220655 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.220656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:53.221223 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:53.221225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.221545 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.221546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:53.221584 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:53.222154 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:53.222158 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:53.223604 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:53.224981 (+ 1377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:53.224986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:53.224988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:53.225173 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:53.225340 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:53.225341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:53.225341 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:53.225394 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:53.225606 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:53.225611 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:53.225650 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 19:04:53.225701 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 19:04:53.225787 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20260925 19:04:53.225926 1963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:53.219411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41996 (local address 127.0.196.254:44651)
0925 19:04:53.219501 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:53.219503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:53.219535 (+ 32us) server_negotiation.cc:407] Connection header received
0925 19:04:53.219606 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:53.219607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:53.219616 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:53.219668 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:53.220106 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.220583 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.221296 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.221481 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.221535 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:53.223678 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:53.223679 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:53.223679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:53.223685 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:53.224910 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:53.225236 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:53.225236 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:53.225237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:53.225284 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:53.225445 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:53.225445 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:53.225445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:53.225553 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:53.225602 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:53.225729 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 19:04:53.225840 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":69}
I20260925 19:04:53.228163 1902 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:41996:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 19:04:53.228355 1902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:04:53.229811 2039 tablet_service.cc:1467] Processing CreateTablet for tablet 3d9d11886fe5400ca61f9db7d1e644b3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aa50a6e178cf4b14b956581b768f596e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 19:04:53.229828 2319 tablet_service.cc:1467] Processing CreateTablet for tablet 3d9d11886fe5400ca61f9db7d1e644b3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aa50a6e178cf4b14b956581b768f596e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 19:04:53.229880 2179 tablet_service.cc:1467] Processing CreateTablet for tablet 3d9d11886fe5400ca61f9db7d1e644b3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aa50a6e178cf4b14b956581b768f596e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 19:04:53.229990 2039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d9d11886fe5400ca61f9db7d1e644b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:53.229992 2179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d9d11886fe5400ca61f9db7d1e644b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:53.229990 2319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d9d11886fe5400ca61f9db7d1e644b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:53.231333 2408 tablet_bootstrap.cc:492] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f: Bootstrap starting.
I20260925 19:04:53.231860 2408 tablet_bootstrap.cc:654] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:53.231913 2409 tablet_bootstrap.cc:492] T 3d9d11886fe5400ca61f9db7d1e644b3 P 11546da847ba4de3a92d36e61708a7c4: Bootstrap starting.
I20260925 19:04:53.231981 2410 tablet_bootstrap.cc:492] T 3d9d11886fe5400ca61f9db7d1e644b3 P 82b4db6ddb0b412992bedd2ccde7d695: Bootstrap starting.
I20260925 19:04:53.232333 2410 tablet_bootstrap.cc:654] T 3d9d11886fe5400ca61f9db7d1e644b3 P 82b4db6ddb0b412992bedd2ccde7d695: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:53.232347 2409 tablet_bootstrap.cc:654] T 3d9d11886fe5400ca61f9db7d1e644b3 P 11546da847ba4de3a92d36e61708a7c4: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:53.232580 2408 tablet_bootstrap.cc:492] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f: No bootstrap required, opened a new log
I20260925 19:04:53.232632 2408 ts_tablet_manager.cc:1397] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 19:04:53.232781 2408 raft_consensus.cc:348] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } }
I20260925 19:04:53.232864 2408 raft_consensus.cc:374] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:53.232887 2408 raft_consensus.cc:729] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db90292012d4a61904ea62ffb27dc8f, State: Initialized, Role: FOLLOWER
I20260925 19:04:53.232931 2410 tablet_bootstrap.cc:492] T 3d9d11886fe5400ca61f9db7d1e644b3 P 82b4db6ddb0b412992bedd2ccde7d695: No bootstrap required, opened a new log
I20260925 19:04:53.232937 2408 consensus_queue.cc:260] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } }
I20260925 19:04:53.232970 2410 ts_tablet_manager.cc:1397] T 3d9d11886fe5400ca61f9db7d1e644b3 P 82b4db6ddb0b412992bedd2ccde7d695: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:04:53.233029 2408 ts_tablet_manager.cc:1428] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:53.233074 2410 raft_consensus.cc:348] T 3d9d11886fe5400ca61f9db7d1e644b3 P 82b4db6ddb0b412992bedd2ccde7d695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } }
I20260925 19:04:53.233132 2410 raft_consensus.cc:374] T 3d9d11886fe5400ca61f9db7d1e644b3 P 82b4db6ddb0b412992bedd2ccde7d695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:53.233177 2410 raft_consensus.cc:729] T 3d9d11886fe5400ca61f9db7d1e644b3 P 82b4db6ddb0b412992bedd2ccde7d695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82b4db6ddb0b412992bedd2ccde7d695, State: Initialized, Role: FOLLOWER
I20260925 19:04:53.233225 2410 consensus_queue.cc:260] T 3d9d11886fe5400ca61f9db7d1e644b3 P 82b4db6ddb0b412992bedd2ccde7d695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } }
I20260925 19:04:53.233327 2410 ts_tablet_manager.cc:1428] T 3d9d11886fe5400ca61f9db7d1e644b3 P 82b4db6ddb0b412992bedd2ccde7d695: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:53.233361 2416 tablet_replica.cc:439] T 3d9d11886fe5400ca61f9db7d1e644b3 P 82b4db6ddb0b412992bedd2ccde7d695: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } } }
I20260925 19:04:53.233441 2416 tablet_replica.cc:442] T 3d9d11886fe5400ca61f9db7d1e644b3 P 82b4db6ddb0b412992bedd2ccde7d695: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:04:53.233495 2414 tablet_replica.cc:439] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } } }
I20260925 19:04:53.233563 2414 tablet_replica.cc:442] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:04:53.234040 2409 tablet_bootstrap.cc:492] T 3d9d11886fe5400ca61f9db7d1e644b3 P 11546da847ba4de3a92d36e61708a7c4: No bootstrap required, opened a new log
I20260925 19:04:53.234094 2409 ts_tablet_manager.cc:1397] T 3d9d11886fe5400ca61f9db7d1e644b3 P 11546da847ba4de3a92d36e61708a7c4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:04:53.234246 2409 raft_consensus.cc:348] T 3d9d11886fe5400ca61f9db7d1e644b3 P 11546da847ba4de3a92d36e61708a7c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } }
I20260925 19:04:53.234338 2409 raft_consensus.cc:374] T 3d9d11886fe5400ca61f9db7d1e644b3 P 11546da847ba4de3a92d36e61708a7c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:53.234364 2409 raft_consensus.cc:729] T 3d9d11886fe5400ca61f9db7d1e644b3 P 11546da847ba4de3a92d36e61708a7c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11546da847ba4de3a92d36e61708a7c4, State: Initialized, Role: FOLLOWER
I20260925 19:04:53.234416 2409 consensus_queue.cc:260] T 3d9d11886fe5400ca61f9db7d1e644b3 P 11546da847ba4de3a92d36e61708a7c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } }
I20260925 19:04:53.234514 2409 ts_tablet_manager.cc:1428] T 3d9d11886fe5400ca61f9db7d1e644b3 P 11546da847ba4de3a92d36e61708a7c4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:53.234570 2415 tablet_replica.cc:439] T 3d9d11886fe5400ca61f9db7d1e644b3 P 11546da847ba4de3a92d36e61708a7c4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } } }
I20260925 19:04:53.234671 2415 tablet_replica.cc:442] T 3d9d11886fe5400ca61f9db7d1e644b3 P 11546da847ba4de3a92d36e61708a7c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:04:53.386672 2414 raft_consensus.cc:482] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:04:53.386788 2414 raft_consensus.cc:504] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } }
I20260925 19:04:53.387210 2414 leader_election.cc:288] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82b4db6ddb0b412992bedd2ccde7d695 (127.0.196.194:45535), 11546da847ba4de3a92d36e61708a7c4 (127.0.196.195:39297)
I20260925 19:04:53.390244 2406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:53.387532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:47695 (local address 127.0.196.194:45535)
0925 19:04:53.387607 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:53.387609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:53.387749 (+ 140us) server_negotiation.cc:407] Connection header received
0925 19:04:53.387783 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:53.387785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:53.387796 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:53.387853 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:53.388027 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.388527 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.389513 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.389879 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.389963 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:53.389973 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:53.390124 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 19:04:53.390164 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 19:04:53.390287 2449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:53.387459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:45535 (local address 127.0.196.193:47695)
0925 19:04:53.387610 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:53.387626 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:53.387721 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:53.387869 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:53.387870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:53.387872 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:53.387954 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:53.387955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.388624 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.388625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:53.389413 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:53.389415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.389972 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.389973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:53.390049 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:53.390053 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:04:53.390119 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 19:04:53.390161 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":15,"thread_start_us":77,"threads_started":1}
I20260925 19:04:53.390342 2404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:53.387647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:43783 (local address 127.0.196.195:39297)
0925 19:04:53.387723 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:53.387725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:53.387781 (+ 56us) server_negotiation.cc:407] Connection header received
0925 19:04:53.387879 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:53.387880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:53.387889 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:53.387965 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:53.388124 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.388596 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.389530 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.389906 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.389991 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:53.390000 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:53.390152 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 19:04:53.390272 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 19:04:53.390297 2450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:53.387500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:39297 (local address 127.0.196.193:43783)
0925 19:04:53.387662 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:53.387673 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:53.387794 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:53.387969 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:53.387970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:53.387972 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:53.388051 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:53.388053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.388683 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.388684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:53.389433 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:53.389435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.390002 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.390003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:53.390078 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:53.390081 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:04:53.390147 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 19:04:53.390189 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":72,"threads_started":1}
I20260925 19:04:53.390560 2199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d9d11886fe5400ca61f9db7d1e644b3" candidate_uuid: "3db90292012d4a61904ea62ffb27dc8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82b4db6ddb0b412992bedd2ccde7d695" is_pre_election: true
I20260925 19:04:53.390681 2199 raft_consensus.cc:2454] T 3d9d11886fe5400ca61f9db7d1e644b3 P 82b4db6ddb0b412992bedd2ccde7d695 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3db90292012d4a61904ea62ffb27dc8f in term 0.
I20260925 19:04:53.390808 2339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d9d11886fe5400ca61f9db7d1e644b3" candidate_uuid: "3db90292012d4a61904ea62ffb27dc8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11546da847ba4de3a92d36e61708a7c4" is_pre_election: true
I20260925 19:04:53.390883 2339 raft_consensus.cc:2454] T 3d9d11886fe5400ca61f9db7d1e644b3 P 11546da847ba4de3a92d36e61708a7c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3db90292012d4a61904ea62ffb27dc8f in term 0.
I20260925 19:04:53.390893 1990 leader_election.cc:302] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3db90292012d4a61904ea62ffb27dc8f, 82b4db6ddb0b412992bedd2ccde7d695; no voters:
I20260925 19:04:53.391045 2414 raft_consensus.cc:2781] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:04:53.391098 2414 raft_consensus.cc:482] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:04:53.391112 2414 raft_consensus.cc:3037] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:53.391778 2414 raft_consensus.cc:504] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } }
I20260925 19:04:53.391932 2414 leader_election.cc:288] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f [CANDIDATE]: Term 1 election: Requested vote from peers 82b4db6ddb0b412992bedd2ccde7d695 (127.0.196.194:45535), 11546da847ba4de3a92d36e61708a7c4 (127.0.196.195:39297)
I20260925 19:04:53.392081 2199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d9d11886fe5400ca61f9db7d1e644b3" candidate_uuid: "3db90292012d4a61904ea62ffb27dc8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82b4db6ddb0b412992bedd2ccde7d695"
I20260925 19:04:53.392119 2339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d9d11886fe5400ca61f9db7d1e644b3" candidate_uuid: "3db90292012d4a61904ea62ffb27dc8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11546da847ba4de3a92d36e61708a7c4"
I20260925 19:04:53.392160 2199 raft_consensus.cc:3037] T 3d9d11886fe5400ca61f9db7d1e644b3 P 82b4db6ddb0b412992bedd2ccde7d695 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:53.392211 2339 raft_consensus.cc:3037] T 3d9d11886fe5400ca61f9db7d1e644b3 P 11546da847ba4de3a92d36e61708a7c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:53.392915 2199 raft_consensus.cc:2454] T 3d9d11886fe5400ca61f9db7d1e644b3 P 82b4db6ddb0b412992bedd2ccde7d695 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3db90292012d4a61904ea62ffb27dc8f in term 1.
I20260925 19:04:53.393090 1990 leader_election.cc:302] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3db90292012d4a61904ea62ffb27dc8f, 82b4db6ddb0b412992bedd2ccde7d695; no voters:
I20260925 19:04:53.393199 2414 raft_consensus.cc:2781] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:53.393330 2414 raft_consensus.cc:686] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f [term 1 LEADER]: Becoming Leader. State: Replica: 3db90292012d4a61904ea62ffb27dc8f, State: Running, Role: LEADER
I20260925 19:04:53.393334 2339 raft_consensus.cc:2454] T 3d9d11886fe5400ca61f9db7d1e644b3 P 11546da847ba4de3a92d36e61708a7c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3db90292012d4a61904ea62ffb27dc8f in term 1.
I20260925 19:04:53.393405 2414 consensus_queue.cc:237] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } }
I20260925 19:04:53.393815 2451 tablet_replica.cc:439] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f: TxnStatusTablet state changed. Reason: New leader 3db90292012d4a61904ea62ffb27dc8f. Latest consensus state: current_term: 1 leader_uuid: "3db90292012d4a61904ea62ffb27dc8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } } }
I20260925 19:04:53.393935 2451 tablet_replica.cc:442] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f: This TxnStatusTablet replica's current role is: LEADER
I20260925 19:04:53.394178 2453 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 19:04:53.394304 2453 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 19:04:53.394214 1901 catalog_manager.cc:5495] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3db90292012d4a61904ea62ffb27dc8f (127.0.196.193). New cstate: current_term: 1 leader_uuid: "3db90292012d4a61904ea62ffb27dc8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } health_report { overall_health: UNKNOWN } } }
I20260925 19:04:53.417968 2445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:53.415350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:36747 (local address 127.0.0.1:48170)
0925 19:04:53.415454 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:53.415467 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:53.415519 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:53.415707 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:53.415708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:53.415710 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:04:53.415763 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:53.415766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.416469 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.416470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:53.417053 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:53.417055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.417424 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.417425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:53.417501 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:53.417524 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:04:53.417798 (+ 274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:04:53.417807 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 19:04:53.417849 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 19:04:53.417900 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 19:04:53.417963 2407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:53.415425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48170 (local address 127.0.196.193:36747)
0925 19:04:53.415495 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:53.415497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:53.415507 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:04:53.415610 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:53.415611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:53.415630 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:53.415699 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:53.415817 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.416373 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.417155 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.417351 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.417427 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:53.417632 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:53.417732 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:53.417815 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:53.417863 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 19:04:53.417882 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 19:04:53.419137 2339 raft_consensus.cc:1261] T 3d9d11886fe5400ca61f9db7d1e644b3 P 11546da847ba4de3a92d36e61708a7c4 [term 1 FOLLOWER]: Refusing update from remote peer 3db90292012d4a61904ea62ffb27dc8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:53.419224 2199 raft_consensus.cc:1261] T 3d9d11886fe5400ca61f9db7d1e644b3 P 82b4db6ddb0b412992bedd2ccde7d695 [term 1 FOLLOWER]: Refusing update from remote peer 3db90292012d4a61904ea62ffb27dc8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:53.419416 2451 consensus_queue.cc:1035] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:53.419553 2414 consensus_queue.cc:1035] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:53.420158 2416 tablet_replica.cc:439] T 3d9d11886fe5400ca61f9db7d1e644b3 P 82b4db6ddb0b412992bedd2ccde7d695: TxnStatusTablet state changed. Reason: New leader 3db90292012d4a61904ea62ffb27dc8f. Latest consensus state: current_term: 1 leader_uuid: "3db90292012d4a61904ea62ffb27dc8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } } }
I20260925 19:04:53.420258 2416 tablet_replica.cc:442] T 3d9d11886fe5400ca61f9db7d1e644b3 P 82b4db6ddb0b412992bedd2ccde7d695: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:04:53.420682 2415 tablet_replica.cc:439] T 3d9d11886fe5400ca61f9db7d1e644b3 P 11546da847ba4de3a92d36e61708a7c4: TxnStatusTablet state changed. Reason: New leader 3db90292012d4a61904ea62ffb27dc8f. Latest consensus state: current_term: 1 leader_uuid: "3db90292012d4a61904ea62ffb27dc8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } } }
I20260925 19:04:53.420794 2415 tablet_replica.cc:442] T 3d9d11886fe5400ca61f9db7d1e644b3 P 11546da847ba4de3a92d36e61708a7c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:04:53.421139 2414 tablet_replica.cc:439] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3db90292012d4a61904ea62ffb27dc8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } } }
I20260925 19:04:53.421224 2414 tablet_replica.cc:442] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f: This TxnStatusTablet replica's current role is: LEADER
I20260925 19:04:53.421424 2422 mvcc.cc:205] Tried to move back new op lower bound from 7333327230642638848 to 7333327230540144640. Current Snapshot: MvccSnapshot[applied={T|T < 7333327230642638848 or (T in {7333327230642638848})}]
I20260925 19:04:53.421676 2421 mvcc.cc:205] Tried to move back new op lower bound from 7333327230642638848 to 7333327230540144640. Current Snapshot: MvccSnapshot[applied={T|T < 7333327230642638848}]
I20260925 19:04:53.421747 2416 tablet_replica.cc:439] T 3d9d11886fe5400ca61f9db7d1e644b3 P 82b4db6ddb0b412992bedd2ccde7d695: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3db90292012d4a61904ea62ffb27dc8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } } }
I20260925 19:04:53.421828 2416 tablet_replica.cc:442] T 3d9d11886fe5400ca61f9db7d1e644b3 P 82b4db6ddb0b412992bedd2ccde7d695: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:04:53.422255 2414 tablet_replica.cc:439] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3db90292012d4a61904ea62ffb27dc8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } } }
I20260925 19:04:53.422338 2414 tablet_replica.cc:442] T 3d9d11886fe5400ca61f9db7d1e644b3 P 3db90292012d4a61904ea62ffb27dc8f: This TxnStatusTablet replica's current role is: LEADER
I20260925 19:04:53.422714 2415 tablet_replica.cc:439] T 3d9d11886fe5400ca61f9db7d1e644b3 P 11546da847ba4de3a92d36e61708a7c4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3db90292012d4a61904ea62ffb27dc8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 } } peers { permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 } } peers { permanent_uuid: "11546da847ba4de3a92d36e61708a7c4" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39297 } } }
I20260925 19:04:53.422801 2415 tablet_replica.cc:442] T 3d9d11886fe5400ca61f9db7d1e644b3 P 11546da847ba4de3a92d36e61708a7c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:04:53.426324 2404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:53.423637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39956 (local address 127.0.196.195:39297)
0925 19:04:53.423677 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:53.423678 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:53.423687 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:04:53.423749 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:53.423749 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:53.423766 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:53.423854 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:53.424008 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.424510 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.425576 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.425766 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.425829 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:53.425991 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:53.426094 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:53.426149 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:53.426217 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 19:04:53.426235 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:04:53.431281 2407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:53.428912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:53809 (local address 127.0.196.193:36747)
0925 19:04:53.428981 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:53.428983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:53.429016 (+ 33us) server_negotiation.cc:407] Connection header received
0925 19:04:53.429081 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:53.429082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:53.429092 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:53.429146 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:53.429286 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.429760 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.430511 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.430780 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.430854 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:53.431034 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:53.431066 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:53.431117 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:53.431182 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 19:04:53.431202 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 19:04:53.431281 2391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:53.428855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:36747 (local address 127.0.196.195:53809)
0925 19:04:53.428963 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:53.428975 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:53.429026 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:53.429149 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:53.429150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:53.429150 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:04:53.429208 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:53.429210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.429848 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.429850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:53.430420 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:53.430421 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.430854 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.430855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:53.430927 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:53.430940 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:04:53.431125 (+ 185us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:04:53.431127 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:04:53.431177 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 19:04:53.431216 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 19:04:53.433503 2199 raft_consensus.cc:1261] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 82b4db6ddb0b412992bedd2ccde7d695 [term 1 FOLLOWER]: Refusing update from remote peer 11546da847ba4de3a92d36e61708a7c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:53.433558 2059 raft_consensus.cc:1261] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 3db90292012d4a61904ea62ffb27dc8f [term 1 FOLLOWER]: Refusing update from remote peer 11546da847ba4de3a92d36e61708a7c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:53.433848 2438 consensus_queue.cc:1035] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db90292012d4a61904ea62ffb27dc8f" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:53.434027 2438 consensus_queue.cc:1035] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82b4db6ddb0b412992bedd2ccde7d695" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:53.444715 2404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:53.441166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:37033 (local address 127.0.196.195:39297)
0925 19:04:53.441210 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:53.441212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:53.441467 (+ 255us) server_negotiation.cc:407] Connection header received
0925 19:04:53.441554 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:53.441555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:53.441567 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:53.441621 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:53.441759 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.442394 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.443707 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.444090 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.444160 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:53.444424 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:53.444457 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:53.444515 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:53.444617 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 19:04:53.444637 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:04:53.444792 2464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:53.440701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:39297 (local address 127.0.196.193:37033)
0925 19:04:53.441394 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:53.441407 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:53.441471 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:53.441633 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:53.441633 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:53.441635 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:04:53.441701 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:53.441703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.442462 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.442463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:53.443618 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:53.443622 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.444197 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.444198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:53.444301 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:53.444318 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:04:53.444529 (+ 211us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:04:53.444531 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:04:53.444607 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 19:04:53.444676 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":636,"threads_started":1}
I20260925 19:04:53.455390 1900 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:04:53.455524 1900 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:04:53.456172 1900 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4fede94c0b9b47c3a3b92e5d8df187cb: Sending DeleteTablet for 3 replicas of tablet 7620dec3ee104dfd8ffe7fe2652de9f7
I20260925 19:04:53.456542 2179 tablet_service.cc:1514] Processing DeleteTablet for tablet 7620dec3ee104dfd8ffe7fe2652de9f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:53 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:35424
I20260925 19:04:53.456542 2039 tablet_service.cc:1514] Processing DeleteTablet for tablet 7620dec3ee104dfd8ffe7fe2652de9f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:53 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:39954
I20260925 19:04:53.456638 2432 tablet_replica.cc:330] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 82b4db6ddb0b412992bedd2ccde7d695: stopping tablet replica
I20260925 19:04:53.456655 2431 tablet_replica.cc:330] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 3db90292012d4a61904ea62ffb27dc8f: stopping tablet replica
I20260925 19:04:53.456694 2432 raft_consensus.cc:2229] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 82b4db6ddb0b412992bedd2ccde7d695 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:04:53.456732 2432 raft_consensus.cc:2258] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 82b4db6ddb0b412992bedd2ccde7d695 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:53.456725 2431 raft_consensus.cc:2229] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 3db90292012d4a61904ea62ffb27dc8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:04:53.456774 2431 raft_consensus.cc:2258] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 3db90292012d4a61904ea62ffb27dc8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:53.457116 2431 ts_tablet_manager.cc:1905] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 3db90292012d4a61904ea62ffb27dc8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:53.457607 2432 ts_tablet_manager.cc:1905] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 82b4db6ddb0b412992bedd2ccde7d695: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:53.458845 2431 ts_tablet_manager.cc:1918] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 3db90292012d4a61904ea62ffb27dc8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:04:53.458909 2431 log.cc:1196] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 3db90292012d4a61904ea62ffb27dc8f: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-0/wal/wals/7620dec3ee104dfd8ffe7fe2652de9f7
I20260925 19:04:53.459019 2432 ts_tablet_manager.cc:1918] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 82b4db6ddb0b412992bedd2ccde7d695: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:04:53.459077 2432 log.cc:1196] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 82b4db6ddb0b412992bedd2ccde7d695: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-1/wal/wals/7620dec3ee104dfd8ffe7fe2652de9f7
I20260925 19:04:53.459128 2431 ts_tablet_manager.cc:1939] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 3db90292012d4a61904ea62ffb27dc8f: Deleting consensus metadata
I20260925 19:04:53.459307 2432 ts_tablet_manager.cc:1939] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 82b4db6ddb0b412992bedd2ccde7d695: Deleting consensus metadata
I20260925 19:04:53.459515 1886 catalog_manager.cc:4841] TS 3db90292012d4a61904ea62ffb27dc8f (127.0.196.193:36747): tablet 7620dec3ee104dfd8ffe7fe2652de9f7 (table test-table [id=09f826bc48bc4f78ae8694f0417a86df]) successfully deleted
I20260925 19:04:53.459721 1886 catalog_manager.cc:4841] TS 82b4db6ddb0b412992bedd2ccde7d695 (127.0.196.194:45535): tablet 7620dec3ee104dfd8ffe7fe2652de9f7 (table test-table [id=09f826bc48bc4f78ae8694f0417a86df]) successfully deleted
I20260925 19:04:53.459869 2319 tablet_service.cc:1514] Processing DeleteTablet for tablet 7620dec3ee104dfd8ffe7fe2652de9f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:53 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:54780
I20260925 19:04:53.459980 2433 tablet_replica.cc:330] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4: stopping tablet replica
I20260925 19:04:53.460040 2433 raft_consensus.cc:2229] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:04:53.460137 2433 raft_consensus.cc:2258] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:53.460487 2433 ts_tablet_manager.cc:1905] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:53.462059 2433 ts_tablet_manager.cc:1918] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:04:53.462121 2433 log.cc:1196] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790363087086945-787-0/minicluster-data/ts-2/wal/wals/7620dec3ee104dfd8ffe7fe2652de9f7
I20260925 19:04:53.462330 2433 ts_tablet_manager.cc:1939] T 7620dec3ee104dfd8ffe7fe2652de9f7 P 11546da847ba4de3a92d36e61708a7c4: Deleting consensus metadata
I20260925 19:04:53.462809 1888 catalog_manager.cc:4841] TS 11546da847ba4de3a92d36e61708a7c4 (127.0.196.195:39297): tablet 7620dec3ee104dfd8ffe7fe2652de9f7 (table test-table [id=09f826bc48bc4f78ae8694f0417a86df]) successfully deleted
Sep 25 19:04:53 dist-test-slave-rn98 krb5kdc[1857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363092, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.196.193@KRBTEST.COM
I20260925 19:04:53.469251 2407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:53.459668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48184 (local address 127.0.196.193:36747)
0925 19:04:53.459714 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:53.459717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:53.460209 (+ 492us) server_negotiation.cc:407] Connection header received
0925 19:04:53.460312 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:53.460313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:53.460324 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:53.460383 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:53.461012 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.461488 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.462296 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.462607 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.462692 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:53.466023 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:53.466033 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:53.466033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:53.466041 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:53.468002 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:53.468399 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:53.468402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:53.468403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:53.468468 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:53.468675 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:53.468676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:53.468676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:53.468835 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:53.468995 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:53.469060 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 19:04:53.469142 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 19:04:53.469632 2099 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48184
I20260925 19:04:53.476737 1963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:53.470578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53750 (local address 127.0.196.254:44651)
0925 19:04:53.470651 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:53.470654 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:53.470818 (+ 164us) server_negotiation.cc:407] Connection header received
0925 19:04:53.470893 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:53.470894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:53.470904 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:53.470959 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:53.471466 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.471957 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.472658 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:53.472874 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:53.472947 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:53.474139 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:53.474140 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:53.474140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:53.474148 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:53.475590 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:53.475954 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:53.475955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:53.475955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:53.476013 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:53.476223 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:53.476224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:53.476224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:53.476340 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:53.476448 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:53.476555 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 19:04:53.476631 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 19:04:53.477068 1900 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53750
I20260925 19:04:53.477720 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 1974
I20260925 19:04:53.484268 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 2114
I20260925 19:04:53.491298 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 2254
I20260925 19:04:53.496460 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 1870
2026-09-25T19:04:53Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2433 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:04:53 dist-test-slave-rn98 krb5kdc[2474](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:04:53 dist-test-slave-rn98 krb5kdc[2474](info): set up 2 sockets
Sep 25 19:04:53 dist-test-slave-rn98 krb5kdc[2474](info): commencing operation
krb5kdc: starting...
W20260925 19:04:54.606564 787 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:04:54 dist-test-slave-rn98 krb5kdc[2474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:04:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:04:54.730137 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:37393
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:35305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:37393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:54.812366 2487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:54.812649 2487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:54.812686 2487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:54.813946 2487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:54.814028 2487 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 19:04:54.814050 2487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:54.814072 2487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:54.814100 2487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:54.814131 2487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:54.815820 2487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:35305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:37393
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:37393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:54.816120 2487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:54.816385 2487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:54.821239 2493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:54.821319 2492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:54.821245 2495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:54.821619 2487 server_base.cc:1034] running on GCE node
I20260925 19:04:54.821882 2487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:54.822243 2487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:54.823437 2487 hybrid_clock.cc:648] HybridClock initialized: now 1790363094823414 us; error 43 us; skew 500 ppm
Sep 25 19:04:54 dist-test-slave-rn98 krb5kdc[2474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:54.826198 2487 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:04:54.826782 2487 webserver.cc:460] Webserver started at http://127.0.196.254:46361/ using document root <none> and password file <none>
I20260925 19:04:54.827035 2487 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:54.827111 2487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:54.827286 2487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:54.828498 2487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "9e404699963a401481b86db6814c2299"
format_stamp: "Formatted at 2026-09-25 19:04:54 on dist-test-slave-rn98"
server_key: "dcd43fd3f5cfe41be5df8b0908118b0f"
server_key_iv: "af75524846fefc1dc4a2b602e774cab4"
server_key_version: "encryptionkey@0"
I20260925 19:04:54.828910 2487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "9e404699963a401481b86db6814c2299"
format_stamp: "Formatted at 2026-09-25 19:04:54 on dist-test-slave-rn98"
server_key: "dcd43fd3f5cfe41be5df8b0908118b0f"
server_key_iv: "af75524846fefc1dc4a2b602e774cab4"
server_key_version: "encryptionkey@0"
I20260925 19:04:54.830386 2487 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:04:54.831355 2502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:54.831532 2487 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:04:54.831643 2487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "9e404699963a401481b86db6814c2299"
format_stamp: "Formatted at 2026-09-25 19:04:54 on dist-test-slave-rn98"
server_key: "dcd43fd3f5cfe41be5df8b0908118b0f"
server_key_iv: "af75524846fefc1dc4a2b602e774cab4"
server_key_version: "encryptionkey@0"
I20260925 19:04:54.831737 2487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:54.847730 2487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:54.850167 2487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:54.850381 2487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:54.855831 2487 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:37393
I20260925 19:04:54.855880 2564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:37393 every 8 connection(s)
I20260925 19:04:54.856292 2487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:54.856940 2565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:54.859920 2565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e404699963a401481b86db6814c2299: Bootstrap starting.
I20260925 19:04:54.860615 2565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e404699963a401481b86db6814c2299: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:54.860940 2565 log.cc:824] T 00000000000000000000000000000000 P 9e404699963a401481b86db6814c2299: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:54.861800 2565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e404699963a401481b86db6814c2299: No bootstrap required, opened a new log
I20260925 19:04:54.864338 2565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e404699963a401481b86db6814c2299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e404699963a401481b86db6814c2299" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37393 } }
I20260925 19:04:54.864506 2565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e404699963a401481b86db6814c2299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:54.864530 2565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e404699963a401481b86db6814c2299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e404699963a401481b86db6814c2299, State: Initialized, Role: FOLLOWER
I20260925 19:04:54.864727 2565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e404699963a401481b86db6814c2299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e404699963a401481b86db6814c2299" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37393 } }
I20260925 19:04:54.864810 2565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e404699963a401481b86db6814c2299 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:04:54.864859 2565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e404699963a401481b86db6814c2299 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:04:54.864915 2565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e404699963a401481b86db6814c2299 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:54.865294 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 2487
I20260925 19:04:54.865413 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:04:54.865579 787 external_mini_cluster.cc:1428] Setting key f6fe15f9dfe5ce31cff5a123223ba125
I20260925 19:04:54.865600 2565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e404699963a401481b86db6814c2299 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e404699963a401481b86db6814c2299" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37393 } }
I20260925 19:04:54.865742 2565 leader_election.cc:302] T 00000000000000000000000000000000 P 9e404699963a401481b86db6814c2299 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e404699963a401481b86db6814c2299; no voters:
I20260925 19:04:54.865969 2565 leader_election.cc:288] T 00000000000000000000000000000000 P 9e404699963a401481b86db6814c2299 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:04:54.866040 2568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e404699963a401481b86db6814c2299 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:54.866191 2568 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e404699963a401481b86db6814c2299 [term 1 LEADER]: Becoming Leader. State: Replica: 9e404699963a401481b86db6814c2299, State: Running, Role: LEADER
I20260925 19:04:54.866283 2565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e404699963a401481b86db6814c2299 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:54.866305 2568 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e404699963a401481b86db6814c2299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e404699963a401481b86db6814c2299" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37393 } }
I20260925 19:04:54.866734 2569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e404699963a401481b86db6814c2299 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e404699963a401481b86db6814c2299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e404699963a401481b86db6814c2299" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37393 } } }
I20260925 19:04:54.866823 2569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e404699963a401481b86db6814c2299 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:54.866966 2569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e404699963a401481b86db6814c2299 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e404699963a401481b86db6814c2299. Latest consensus state: current_term: 1 leader_uuid: "9e404699963a401481b86db6814c2299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e404699963a401481b86db6814c2299" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37393 } } }
I20260925 19:04:54.867033 2569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e404699963a401481b86db6814c2299 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:54.867188 2577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:04:54.867949 2577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:54.869510 2577 catalog_manager.cc:1347] Generated new cluster ID: 4c784eaf740f4364bfcf36b534317bdb
I20260925 19:04:54.869565 2577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:04:54 dist-test-slave-rn98 krb5kdc[2474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:04:54.879138 2572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:54.866144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36082 (local address 127.0.196.254:37393)
0925 19:04:54.866745 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:54.866768 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:54.866809 (+ 41us) server_negotiation.cc:407] Connection header received
0925 19:04:54.866968 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:54.866978 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:54.867065 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:54.867893 (+ 828us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:54.868146 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:54.869840 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:54.870920 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:54.871609 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:54.871738 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:54.874761 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:54.874766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:54.874766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:54.874779 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:54.877305 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:54.877756 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:54.877759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:54.877760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:54.877889 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:54.878128 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:54.878129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:54.878129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:54.878436 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:54.878590 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:54.878767 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 19:04:54.878905 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":76,"threads_started":1}
I20260925 19:04:54.898550 2577 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:54.899206 2577 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:54.904881 2577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e404699963a401481b86db6814c2299: Generated new TSK 0
I20260925 19:04:54.905100 2577 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
I20260925 19:04:54.934302 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:0
--local_ip_for_outbound_sockets=127.0.196.193
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:37393
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:35305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:55.018394 2591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:55.018684 2591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:55.018739 2591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:55.019920 2591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:55.019959 2591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:55.019979 2591 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 19:04:55.019991 2591 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 19:04:55.020008 2591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:04:55.021430 2591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:35305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.196.254:37393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:55.021629 2591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:55.021813 2591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:55.026978 2596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:55.026970 2599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:55.026974 2597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:55.027119 2591 server_base.cc:1034] running on GCE node
I20260925 19:04:55.027484 2591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:55.027731 2591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:55.028904 2591 hybrid_clock.cc:648] HybridClock initialized: now 1790363095028898 us; error 44 us; skew 500 ppm
Sep 25 19:04:55 dist-test-slave-rn98 krb5kdc[2474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:55.031615 2591 init.cc:377] Logged in from keytab as kudu/127.0.196.193@KRBTEST.COM (short username kudu)
I20260925 19:04:55.032050 2591 webserver.cc:460] Webserver started at http://127.0.196.193:35957/ using document root <none> and password file <none>
I20260925 19:04:55.032302 2591 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:55.032347 2591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:55.032497 2591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:55.033557 2591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-0/data/instance:
uuid: "ce330baa60b343afac700000a3f99c82"
format_stamp: "Formatted at 2026-09-25 19:04:55 on dist-test-slave-rn98"
server_key: "e62f5597d29fb3b9edb7158f932ad275"
server_key_iv: "b7818729bb88d1eb82670b9f830f1841"
server_key_version: "encryptionkey@0"
I20260925 19:04:55.033913 2591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-0/wal/instance:
uuid: "ce330baa60b343afac700000a3f99c82"
format_stamp: "Formatted at 2026-09-25 19:04:55 on dist-test-slave-rn98"
server_key: "e62f5597d29fb3b9edb7158f932ad275"
server_key_iv: "b7818729bb88d1eb82670b9f830f1841"
server_key_version: "encryptionkey@0"
I20260925 19:04:55.035414 2591 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:04:55.036337 2606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:55.036557 2591 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:04:55.036695 2591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "ce330baa60b343afac700000a3f99c82"
format_stamp: "Formatted at 2026-09-25 19:04:55 on dist-test-slave-rn98"
server_key: "e62f5597d29fb3b9edb7158f932ad275"
server_key_iv: "b7818729bb88d1eb82670b9f830f1841"
server_key_version: "encryptionkey@0"
I20260925 19:04:55.036784 2591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:55.056380 2591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:55.058373 2591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:55.058552 2591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:55.059150 2591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:55.059226 2591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:55.059273 2591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:55.059304 2591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:55.067377 2591 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:45403
I20260925 19:04:55.067447 2721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:45403 every 8 connection(s)
Sep 25 19:04:55 dist-test-slave-rn98 krb5kdc[2474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:04:55.067865 2591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
I20260925 19:04:55.069464 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 2591
I20260925 19:04:55.069641 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-0/wal/instance
I20260925 19:04:55.069846 787 external_mini_cluster.cc:1428] Setting key cc057fbdf8b59993c79d3fa5b900f85f
I20260925 19:04:55.071904 2572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.059681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:57549 (local address 127.0.196.254:37393)
0925 19:04:55.059739 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:55.059741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:55.060005 (+ 264us) server_negotiation.cc:407] Connection header received
0925 19:04:55.060559 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:55.060560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:55.060577 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:55.060638 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:55.061584 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.062588 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.064609 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.064974 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.065041 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.068411 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:55.068413 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:55.068418 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:55.068426 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:55.070252 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:55.070800 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:55.070801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:55.070801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:55.070864 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:55.071176 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:55.071176 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:55.071177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:55.071291 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:55.071344 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:55.071745 (+ 401us) server_negotiation.cc:299] Negotiation successful
0925 19:04:55.071840 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 19:04:55.072252 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.059673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:37393 (local address 127.0.196.193:57549)
0925 19:04:55.059873 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:55.059910 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:55.060443 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:55.060712 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:55.060716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:55.061101 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:55.061452 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.061460 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.062760 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.062762 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.064485 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.064489 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.065079 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.065080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.065297 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.066065 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:55.066074 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:55.068283 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:55.070327 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:55.070335 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:55.070337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:55.070722 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:55.070939 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:55.070940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:55.070941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:55.071103 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:55.071357 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:55.071363 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:55.071586 (+ 223us) client_negotiation.cc:769] Sending connection context
0925 19:04:55.071735 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 19:04:55.071943 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":113,"threads_started":1}
I20260925 19:04:55.072779 2722 heartbeater.cc:347] Connected to a master server at 127.0.196.254:37393
I20260925 19:04:55.072872 2722 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:55.073091 2722 heartbeater.cc:511] Master 127.0.196.254:37393 requested a full tablet report, sending...
I20260925 19:04:55.073675 2518 ts_manager.cc:194] Registered new tserver with Master: ce330baa60b343afac700000a3f99c82 (127.0.196.193:45403)
I20260925 19:04:55.074651 2518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.193@KRBTEST.COM'} at 127.0.196.193:57549
WARNING: no policy specified for kudu/127.0.196.194@KRBTEST.COM; defaulting to no policy
I20260925 19:04:55.081449 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.074856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:37393 (local address 127.0.196.193:37223)
0925 19:04:55.075157 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:55.075169 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:55.075242 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:55.075360 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:55.075360 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:55.075589 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:55.075667 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.075669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.076370 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.076371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.077278 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.077282 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.077630 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.077631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.077716 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.078175 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:55.078182 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:55.078587 (+ 405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:55.080455 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:55.080457 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:55.080457 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:55.080640 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:55.080847 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:55.080848 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:55.080848 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:55.080899 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:55.081114 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:55.081115 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:55.081154 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 19:04:55.081212 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 19:04:55.081351 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":116,"threads_started":1}
I20260925 19:04:55.081470 2572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.075077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:37223 (local address 127.0.196.254:37393)
0925 19:04:55.075119 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:55.075121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:55.075243 (+ 122us) server_negotiation.cc:407] Connection header received
0925 19:04:55.075284 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:55.075284 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:55.075294 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:55.075354 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:55.075754 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.076228 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.077374 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.077554 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.077603 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.078705 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:55.078707 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:55.078707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:55.078714 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:55.080356 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:55.080709 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:55.080710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:55.080710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:55.080772 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:55.080952 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:55.080953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:55.080953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:55.081060 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:55.081113 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:55.081229 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 19:04:55.081389 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
I20260925 19:04:55.099922 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:0
--local_ip_for_outbound_sockets=127.0.196.194
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:37393
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:35305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:55.181933 2731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:55.182191 2731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:55.182235 2731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:55.183444 2731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:55.183519 2731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:55.183554 2731 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 19:04:55.183581 2731 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 19:04:55.183617 2731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:04:55.185098 2731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:35305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.196.254:37393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:55.185384 2731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:55.185619 2731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:04:55.190927 2731 server_base.cc:1034] running on GCE node
W20260925 19:04:55.190889 2739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:55.190874 2737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:55.190879 2736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:55.191295 2731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:55.191550 2731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:55.192691 2731 hybrid_clock.cc:648] HybridClock initialized: now 1790363095192683 us; error 33 us; skew 500 ppm
Sep 25 19:04:55 dist-test-slave-rn98 krb5kdc[2474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:55.195317 2731 init.cc:377] Logged in from keytab as kudu/127.0.196.194@KRBTEST.COM (short username kudu)
I20260925 19:04:55.195770 2731 webserver.cc:460] Webserver started at http://127.0.196.194:45807/ using document root <none> and password file <none>
I20260925 19:04:55.195995 2731 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:55.196065 2731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:55.196234 2731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:55.197335 2731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-1/data/instance:
uuid: "40adefa7979c40d59796453a098c6a55"
format_stamp: "Formatted at 2026-09-25 19:04:55 on dist-test-slave-rn98"
server_key: "6d0544645095078abc63cc4611ad3e40"
server_key_iv: "ae9336b205d2eae6726a6c9ba0d4f01c"
server_key_version: "encryptionkey@0"
I20260925 19:04:55.197710 2731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-1/wal/instance:
uuid: "40adefa7979c40d59796453a098c6a55"
format_stamp: "Formatted at 2026-09-25 19:04:55 on dist-test-slave-rn98"
server_key: "6d0544645095078abc63cc4611ad3e40"
server_key_iv: "ae9336b205d2eae6726a6c9ba0d4f01c"
server_key_version: "encryptionkey@0"
I20260925 19:04:55.199172 2731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:04:55.200002 2746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:55.200208 2731 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:04:55.200363 2731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "40adefa7979c40d59796453a098c6a55"
format_stamp: "Formatted at 2026-09-25 19:04:55 on dist-test-slave-rn98"
server_key: "6d0544645095078abc63cc4611ad3e40"
server_key_iv: "ae9336b205d2eae6726a6c9ba0d4f01c"
server_key_version: "encryptionkey@0"
I20260925 19:04:55.200448 2731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:55.246810 2731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:55.249351 2731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:55.249536 2731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:55.250084 2731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:55.250146 2731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:55.250182 2731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:55.250218 2731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:55.257545 2731 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:44319
I20260925 19:04:55.257639 2861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:44319 every 8 connection(s)
I20260925 19:04:55.257961 2731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:04:55 dist-test-slave-rn98 krb5kdc[2474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:04:55.261317 2572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.250762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:49915 (local address 127.0.196.254:37393)
0925 19:04:55.250806 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:55.250810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:55.251115 (+ 305us) server_negotiation.cc:407] Connection header received
0925 19:04:55.251630 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:55.251631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:55.251645 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:55.251693 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:55.252512 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.253340 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.255136 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.255466 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.255539 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.258684 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:55.258685 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:55.258686 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:55.258693 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:55.259924 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:55.260380 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:55.260380 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:55.260381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:55.260430 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:55.260777 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:55.260777 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:55.260777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:55.260882 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:55.260935 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:55.261188 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 19:04:55.261252 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:04:55.261610 2755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.250789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:37393 (local address 127.0.196.194:49915)
0925 19:04:55.250963 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:55.251002 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:55.251519 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:55.251770 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:55.251774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:55.252111 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:55.252409 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.252414 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.253450 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.253451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.255037 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.255041 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.255558 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.255559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.255721 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.256508 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:55.256521 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:55.258541 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:55.260005 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:55.260011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:55.260013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:55.260311 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:55.260591 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:55.260592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:55.260592 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:55.260702 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:55.260946 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:55.260950 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:55.261072 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 19:04:55.261162 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 19:04:55.261294 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":96,"threads_started":1}
I20260925 19:04:55.262152 2862 heartbeater.cc:347] Connected to a master server at 127.0.196.254:37393
I20260925 19:04:55.262236 2862 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:55.262428 2862 heartbeater.cc:511] Master 127.0.196.254:37393 requested a full tablet report, sending...
I20260925 19:04:55.262837 2518 ts_manager.cc:194] Registered new tserver with Master: 40adefa7979c40d59796453a098c6a55 (127.0.196.194:44319)
I20260925 19:04:55.263224 2518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.194@KRBTEST.COM'} at 127.0.196.194:49915
I20260925 19:04:55.265654 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 2731
I20260925 19:04:55.265748 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-1/wal/instance
I20260925 19:04:55.265928 787 external_mini_cluster.cc:1428] Setting key 472f6e4e7abf2da09649e66c3b87146a
I20260925 19:04:55.269456 2572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.263348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:46565 (local address 127.0.196.254:37393)
0925 19:04:55.263388 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:55.263390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:55.263533 (+ 143us) server_negotiation.cc:407] Connection header received
0925 19:04:55.263604 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:55.263604 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:55.263614 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:55.263669 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:55.263985 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.264476 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.265177 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.265381 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.265464 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.266552 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:55.266554 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:55.266554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:55.266561 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:55.268257 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:55.268590 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:55.268592 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:55.268592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:55.268648 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:55.268885 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:55.268886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:55.268886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:55.269050 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:55.269126 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:55.269301 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 19:04:55.269393 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:04:55.269542 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.263247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:37393 (local address 127.0.196.194:46565)
0925 19:04:55.263469 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:55.263479 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:55.263547 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:55.263676 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:55.263676 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:55.263858 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:55.263906 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.263908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.264575 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.264576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.265080 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.265082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.265465 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.265466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.265540 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.266030 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:55.266035 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:55.266459 (+ 424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:55.268338 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:55.268339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:55.268340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:55.268515 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:55.268725 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:55.268726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:55.268726 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:55.268787 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:55.269126 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:55.269128 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:55.269180 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 19:04:55.269251 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 19:04:55.269383 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for kudu/127.0.196.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
I20260925 19:04:55.294714 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:0
--local_ip_for_outbound_sockets=127.0.196.195
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:37393
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:35305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:55.377801 2871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:55.378021 2871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:55.378046 2871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:55.379297 2871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:55.379341 2871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:55.379362 2871 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 19:04:55.379374 2871 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 19:04:55.379391 2871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:04:55.380800 2871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:35305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.196.254:37393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:55.381002 2871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:55.381182 2871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:55.386147 2879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:55.386267 2877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:55.386417 2876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:55.386447 2871 server_base.cc:1034] running on GCE node
I20260925 19:04:55.386751 2871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:55.387015 2871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:55.388187 2871 hybrid_clock.cc:648] HybridClock initialized: now 1790363095388131 us; error 72 us; skew 500 ppm
Sep 25 19:04:55 dist-test-slave-rn98 krb5kdc[2474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:55.390797 2871 init.cc:377] Logged in from keytab as kudu/127.0.196.195@KRBTEST.COM (short username kudu)
I20260925 19:04:55.391209 2871 webserver.cc:460] Webserver started at http://127.0.196.195:34269/ using document root <none> and password file <none>
I20260925 19:04:55.391404 2871 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:55.391445 2871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:55.391535 2871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:55.392561 2871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-2/data/instance:
uuid: "941d3d5f8da64b18ac96345644de3869"
format_stamp: "Formatted at 2026-09-25 19:04:55 on dist-test-slave-rn98"
server_key: "8f8f9c010369a6c44881adbeca52208d"
server_key_iv: "d05be69bcee114fef161e98c656d549c"
server_key_version: "encryptionkey@0"
I20260925 19:04:55.392863 2871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-2/wal/instance:
uuid: "941d3d5f8da64b18ac96345644de3869"
format_stamp: "Formatted at 2026-09-25 19:04:55 on dist-test-slave-rn98"
server_key: "8f8f9c010369a6c44881adbeca52208d"
server_key_iv: "d05be69bcee114fef161e98c656d549c"
server_key_version: "encryptionkey@0"
I20260925 19:04:55.394145 2871 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:04:55.394942 2886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:55.395103 2871 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:04:55.395197 2871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "941d3d5f8da64b18ac96345644de3869"
format_stamp: "Formatted at 2026-09-25 19:04:55 on dist-test-slave-rn98"
server_key: "8f8f9c010369a6c44881adbeca52208d"
server_key_iv: "d05be69bcee114fef161e98c656d549c"
server_key_version: "encryptionkey@0"
I20260925 19:04:55.395259 2871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:55.408367 2871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:55.410873 2871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:55.411003 2871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:55.411487 2871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:55.411530 2871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:55.411556 2871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:55.411568 2871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:55.418668 2871 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:33983
I20260925 19:04:55.418745 3001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:33983 every 8 connection(s)
Sep 25 19:04:55 dist-test-slave-rn98 krb5kdc[2474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:04:55.419092 2871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
I20260925 19:04:55.419612 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 2871
I20260925 19:04:55.419705 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-2/wal/instance
I20260925 19:04:55.419873 787 external_mini_cluster.cc:1428] Setting key a5a5b62b29438cee62ab8794e0780aa7
I20260925 19:04:55.422104 2572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.412125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:59945 (local address 127.0.196.254:37393)
0925 19:04:55.412266 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:55.412269 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:55.412348 (+ 79us) server_negotiation.cc:407] Connection header received
0925 19:04:55.412862 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:55.412864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:55.412879 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:55.412968 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:55.413892 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.414800 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.416444 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.416761 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.416845 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.419410 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:55.419412 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:55.419412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:55.419421 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:55.420955 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:55.421360 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:55.421361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:55.421361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:55.421410 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:55.421593 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:55.421594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:55.421594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:55.421693 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:55.421743 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:55.421981 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 19:04:55.422042 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260925 19:04:55.422276 2895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.412033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:37393 (local address 127.0.196.195:59945)
0925 19:04:55.412193 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:55.412236 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:55.412741 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:55.413041 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:55.413044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:55.413447 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:55.413759 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.413767 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.414912 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.414914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.416343 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.416348 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.416863 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.416864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.416967 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.417595 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:55.417609 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:55.419280 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:55.421027 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:55.421033 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:55.421034 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:55.421291 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:55.421455 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:55.421455 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:55.421456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:55.421544 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:55.421750 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:55.421754 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:55.421870 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 19:04:55.421953 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 19:04:55.422074 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":94,"threads_started":1}
I20260925 19:04:55.422775 3002 heartbeater.cc:347] Connected to a master server at 127.0.196.254:37393
I20260925 19:04:55.422865 3002 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:55.423066 3002 heartbeater.cc:511] Master 127.0.196.254:37393 requested a full tablet report, sending...
I20260925 19:04:55.423478 2519 ts_manager.cc:194] Registered new tserver with Master: 941d3d5f8da64b18ac96345644de3869 (127.0.196.195:33983)
I20260925 19:04:55.423985 2519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.195@KRBTEST.COM'} at 127.0.196.195:59945
I20260925 19:04:55.429397 2572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.423977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:57603 (local address 127.0.196.254:37393)
0925 19:04:55.424117 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:55.424119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:55.424139 (+ 20us) server_negotiation.cc:407] Connection header received
0925 19:04:55.424156 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:55.424157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:55.424165 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:55.424209 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:55.424524 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.424995 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.425702 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.425897 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.425969 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.426886 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:55.426888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:55.426888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:55.426895 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:55.428239 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:55.428604 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:55.428605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:55.428606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:55.428673 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:55.428874 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:55.428874 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:55.428875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:55.428983 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:55.429060 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:55.429227 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 19:04:55.429303 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260925 19:04:55.429412 3008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.423903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:37393 (local address 127.0.196.195:57603)
0925 19:04:55.424035 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:55.424046 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:55.424095 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:55.424213 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:55.424214 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:55.424396 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:55.424447 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.424448 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.425087 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.425088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.425604 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.425605 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.425978 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.425979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.426053 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.426460 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:55.426464 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:55.426788 (+ 324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:55.428351 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:55.428352 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:55.428352 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:55.428528 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:55.428740 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:55.428741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:55.428741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:55.428800 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:55.429050 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:55.429051 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:55.429097 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 19:04:55.429199 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 19:04:55.429305 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":70,"threads_started":1}
I20260925 19:04:55.431520 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 19:04:55 dist-test-slave-rn98 krb5kdc[2474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 19:04:55 dist-test-slave-rn98 krb5kdc[2474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:04:55.456933 2572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.449828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36086 (local address 127.0.196.254:37393)
0925 19:04:55.449897 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:55.449899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:55.450103 (+ 204us) server_negotiation.cc:407] Connection header received
0925 19:04:55.450189 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:55.450190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:55.450202 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:55.450272 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:55.450891 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.451385 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.452079 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.452254 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.452303 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.454496 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:55.454498 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:55.454498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:55.454505 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:55.455976 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:55.456303 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:55.456304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:55.456304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:55.456364 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:55.456533 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:55.456534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:55.456534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:55.456636 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:55.456712 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:55.456796 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 19:04:55.456866 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:04:55.458788 2519 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:04:55.460655 2519 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:04:55.468009 3023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.463811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60204 (local address 127.0.196.195:33983)
0925 19:04:55.463961 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:55.463964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:55.464361 (+ 397us) server_negotiation.cc:407] Connection header received
0925 19:04:55.464404 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:55.464406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:55.464421 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:55.464491 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:55.464691 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.465380 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.466977 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.467591 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.467661 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.467721 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:55.467877 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 19:04:55.467915 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260925 19:04:55.468004 3020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.463699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:33983 (local address 127.0.0.1:60204)
0925 19:04:55.464202 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:55.464211 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:55.464263 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:55.464556 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:55.464557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:55.464559 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:55.464614 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.464615 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.465470 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.465471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.466876 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.466878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.467659 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.467660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.467735 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.467750 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 19:04:55.467854 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 19:04:55.467913 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":5,"thread_start_us":60,"threads_started":1}
I20260925 19:04:55.468335 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.463835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42664 (local address 127.0.196.194:44319)
0925 19:04:55.464011 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:55.464016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:55.464161 (+ 145us) server_negotiation.cc:407] Connection header received
0925 19:04:55.464203 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:55.464205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:55.464221 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:55.464487 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:55.464540 (+ 53us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.465596 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.467002 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.467733 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.468101 (+ 368us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.468173 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:55.468205 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 19:04:55.468246 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":73,"threads_started":1}
I20260925 19:04:55.468384 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.463498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:45403 (local address 127.0.0.1:40256)
0925 19:04:55.463801 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:55.463845 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:55.463923 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:55.464183 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:55.464186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:55.464189 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:55.464271 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.464273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.465648 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.465649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.466867 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.466870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.468146 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.468147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.468226 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.468230 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:04:55.468282 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 19:04:55.468318 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":5,"thread_start_us":55,"threads_started":1}
I20260925 19:04:55.468649 3022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.463541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:44319 (local address 127.0.0.1:42664)
0925 19:04:55.464060 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:55.464069 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:55.464131 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:55.464369 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:55.464369 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:55.464371 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:55.464415 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.464417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.465745 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.465746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.466887 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.466890 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.467850 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.467851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.467917 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.467921 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:04:55.467952 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 19:04:55.467995 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":84,"threads_started":1}
I20260925 19:04:55.468807 2936 tablet_service.cc:1467] Processing CreateTablet for tablet cc94e22bad16422d8fef9a2153c2b173 (DEFAULT_TABLE table=test-table [id=e2a5672b4699415f9291c67ed4c4f62f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:55.468981 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.463606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40256 (local address 127.0.196.193:45403)
0925 19:04:55.463782 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:55.463788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:55.464013 (+ 225us) server_negotiation.cc:407] Connection header received
0925 19:04:55.464061 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:55.464064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:55.464083 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:55.464161 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:55.464360 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.465563 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.466957 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.467789 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.467912 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.467972 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:55.468814 (+ 842us) server_negotiation.cc:299] Negotiation successful
0925 19:04:55.468861 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260925 19:04:55.469208 2936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc94e22bad16422d8fef9a2153c2b173. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:55.471256 2796 tablet_service.cc:1467] Processing CreateTablet for tablet cc94e22bad16422d8fef9a2153c2b173 (DEFAULT_TABLE table=test-table [id=e2a5672b4699415f9291c67ed4c4f62f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:55.471359 2654 tablet_service.cc:1467] Processing CreateTablet for tablet cc94e22bad16422d8fef9a2153c2b173 (DEFAULT_TABLE table=test-table [id=e2a5672b4699415f9291c67ed4c4f62f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:55.471608 2796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc94e22bad16422d8fef9a2153c2b173. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:55.471710 2654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc94e22bad16422d8fef9a2153c2b173. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:55.474598 3026 tablet_bootstrap.cc:492] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82: Bootstrap starting.
I20260925 19:04:55.474828 3027 tablet_bootstrap.cc:492] T cc94e22bad16422d8fef9a2153c2b173 P 40adefa7979c40d59796453a098c6a55: Bootstrap starting.
I20260925 19:04:55.475358 3026 tablet_bootstrap.cc:654] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:55.475358 3027 tablet_bootstrap.cc:654] T cc94e22bad16422d8fef9a2153c2b173 P 40adefa7979c40d59796453a098c6a55: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:55.475714 3026 log.cc:824] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:55.475719 3027 log.cc:824] T cc94e22bad16422d8fef9a2153c2b173 P 40adefa7979c40d59796453a098c6a55: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:55.476140 3025 tablet_bootstrap.cc:492] T cc94e22bad16422d8fef9a2153c2b173 P 941d3d5f8da64b18ac96345644de3869: Bootstrap starting.
I20260925 19:04:55.476502 3027 tablet_bootstrap.cc:492] T cc94e22bad16422d8fef9a2153c2b173 P 40adefa7979c40d59796453a098c6a55: No bootstrap required, opened a new log
I20260925 19:04:55.476585 3027 ts_tablet_manager.cc:1397] T cc94e22bad16422d8fef9a2153c2b173 P 40adefa7979c40d59796453a098c6a55: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:04:55.476799 3026 tablet_bootstrap.cc:492] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82: No bootstrap required, opened a new log
I20260925 19:04:55.476894 3026 ts_tablet_manager.cc:1397] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 19:04:55.477193 3025 tablet_bootstrap.cc:654] T cc94e22bad16422d8fef9a2153c2b173 P 941d3d5f8da64b18ac96345644de3869: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:55.477538 3025 log.cc:824] T cc94e22bad16422d8fef9a2153c2b173 P 941d3d5f8da64b18ac96345644de3869: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:55.478170 3027 raft_consensus.cc:348] T cc94e22bad16422d8fef9a2153c2b173 P 40adefa7979c40d59796453a098c6a55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } }
I20260925 19:04:55.478338 3027 raft_consensus.cc:374] T cc94e22bad16422d8fef9a2153c2b173 P 40adefa7979c40d59796453a098c6a55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:55.478386 3027 raft_consensus.cc:729] T cc94e22bad16422d8fef9a2153c2b173 P 40adefa7979c40d59796453a098c6a55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40adefa7979c40d59796453a098c6a55, State: Initialized, Role: FOLLOWER
I20260925 19:04:55.478547 3027 consensus_queue.cc:260] T cc94e22bad16422d8fef9a2153c2b173 P 40adefa7979c40d59796453a098c6a55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } }
I20260925 19:04:55.478905 3027 ts_tablet_manager.cc:1428] T cc94e22bad16422d8fef9a2153c2b173 P 40adefa7979c40d59796453a098c6a55: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 19:04:55.478842 3026 raft_consensus.cc:348] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } }
I20260925 19:04:55.479017 3026 raft_consensus.cc:374] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:55.479059 3026 raft_consensus.cc:729] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce330baa60b343afac700000a3f99c82, State: Initialized, Role: FOLLOWER
I20260925 19:04:55.479096 2862 heartbeater.cc:503] Master 127.0.196.254:37393 was elected leader, sending a full tablet report...
I20260925 19:04:55.479197 3026 consensus_queue.cc:260] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } }
I20260925 19:04:55.479254 3025 tablet_bootstrap.cc:492] T cc94e22bad16422d8fef9a2153c2b173 P 941d3d5f8da64b18ac96345644de3869: No bootstrap required, opened a new log
I20260925 19:04:55.479333 3025 ts_tablet_manager.cc:1397] T cc94e22bad16422d8fef9a2153c2b173 P 941d3d5f8da64b18ac96345644de3869: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 19:04:55.479491 3026 ts_tablet_manager.cc:1428] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 19:04:55.479504 2722 heartbeater.cc:503] Master 127.0.196.254:37393 was elected leader, sending a full tablet report...
I20260925 19:04:55.481211 3025 raft_consensus.cc:348] T cc94e22bad16422d8fef9a2153c2b173 P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } }
I20260925 19:04:55.481345 3025 raft_consensus.cc:374] T cc94e22bad16422d8fef9a2153c2b173 P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:55.481367 3025 raft_consensus.cc:729] T cc94e22bad16422d8fef9a2153c2b173 P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 941d3d5f8da64b18ac96345644de3869, State: Initialized, Role: FOLLOWER
I20260925 19:04:55.481469 3025 consensus_queue.cc:260] T cc94e22bad16422d8fef9a2153c2b173 P 941d3d5f8da64b18ac96345644de3869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } }
I20260925 19:04:55.481673 3025 ts_tablet_manager.cc:1428] T cc94e22bad16422d8fef9a2153c2b173 P 941d3d5f8da64b18ac96345644de3869: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:04:55.481736 3002 heartbeater.cc:503] Master 127.0.196.254:37393 was elected leader, sending a full tablet report...
W20260925 19:04:55.508579 2863 tablet.cc:2369] T cc94e22bad16422d8fef9a2153c2b173 P 40adefa7979c40d59796453a098c6a55: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:04:55.568722 2723 tablet.cc:2369] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:04:55.637796 3032 raft_consensus.cc:482] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:04:55.637955 3032 raft_consensus.cc:504] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } }
I20260925 19:04:55.638367 3032 leader_election.cc:288] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 941d3d5f8da64b18ac96345644de3869 (127.0.196.195:33983), 40adefa7979c40d59796453a098c6a55 (127.0.196.194:44319)
I20260925 19:04:55.641649 3023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.638684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:45451 (local address 127.0.196.195:33983)
0925 19:04:55.638738 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:55.638740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:55.638825 (+ 85us) server_negotiation.cc:407] Connection header received
0925 19:04:55.638943 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:55.638944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:55.638956 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:55.639039 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:55.639211 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.639683 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.640793 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.641297 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.641388 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.641397 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:55.641543 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 19:04:55.641581 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 19:04:55.641615 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.638783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:58237 (local address 127.0.196.194:44319)
0925 19:04:55.638890 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:55.638893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:55.638964 (+ 71us) server_negotiation.cc:407] Connection header received
0925 19:04:55.639056 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:55.639057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:55.639072 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:55.639133 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:55.639284 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.639763 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.640834 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.641258 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.641342 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.641351 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:55.641493 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 19:04:55.641531 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 19:04:55.641657 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.638659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:44319 (local address 127.0.196.193:58237)
0925 19:04:55.638885 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:55.638898 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:55.638984 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:55.639136 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:55.639137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:55.639139 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:55.639211 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.639217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.639865 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.639866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.640745 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.640748 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.641338 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.641339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.641418 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.641422 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:04:55.641488 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 19:04:55.641531 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":62,"threads_started":1}
I20260925 19:04:55.641681 3034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.638570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:33983 (local address 127.0.196.193:45451)
0925 19:04:55.638743 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:55.638756 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:55.638863 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:55.639047 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:55.639049 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:55.639051 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:55.639126 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.639128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.639768 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.639769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.640678 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.640681 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.641390 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.641390 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.641466 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.641469 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:04:55.641538 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 19:04:55.641579 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":60,"thread_start_us":119,"threads_started":1}
I20260925 19:04:55.642028 2816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc94e22bad16422d8fef9a2153c2b173" candidate_uuid: "ce330baa60b343afac700000a3f99c82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40adefa7979c40d59796453a098c6a55" is_pre_election: true
I20260925 19:04:55.642234 2816 raft_consensus.cc:2454] T cc94e22bad16422d8fef9a2153c2b173 P 40adefa7979c40d59796453a098c6a55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce330baa60b343afac700000a3f99c82 in term 0.
I20260925 19:04:55.642434 2956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc94e22bad16422d8fef9a2153c2b173" candidate_uuid: "ce330baa60b343afac700000a3f99c82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "941d3d5f8da64b18ac96345644de3869" is_pre_election: true
I20260925 19:04:55.642580 2956 raft_consensus.cc:2454] T cc94e22bad16422d8fef9a2153c2b173 P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce330baa60b343afac700000a3f99c82 in term 0.
I20260925 19:04:55.642567 2610 leader_election.cc:302] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40adefa7979c40d59796453a098c6a55, ce330baa60b343afac700000a3f99c82; no voters:
I20260925 19:04:55.642733 3032 raft_consensus.cc:2781] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:04:55.642786 3032 raft_consensus.cc:482] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:04:55.642818 3032 raft_consensus.cc:3037] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:55.643509 3032 raft_consensus.cc:504] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } }
I20260925 19:04:55.643667 3032 leader_election.cc:288] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82 [CANDIDATE]: Term 1 election: Requested vote from peers 941d3d5f8da64b18ac96345644de3869 (127.0.196.195:33983), 40adefa7979c40d59796453a098c6a55 (127.0.196.194:44319)
I20260925 19:04:55.643824 2816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc94e22bad16422d8fef9a2153c2b173" candidate_uuid: "ce330baa60b343afac700000a3f99c82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40adefa7979c40d59796453a098c6a55"
I20260925 19:04:55.643849 2956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc94e22bad16422d8fef9a2153c2b173" candidate_uuid: "ce330baa60b343afac700000a3f99c82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "941d3d5f8da64b18ac96345644de3869"
I20260925 19:04:55.643941 2816 raft_consensus.cc:3037] T cc94e22bad16422d8fef9a2153c2b173 P 40adefa7979c40d59796453a098c6a55 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:55.643936 2956 raft_consensus.cc:3037] T cc94e22bad16422d8fef9a2153c2b173 P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:55.644770 2956 raft_consensus.cc:2454] T cc94e22bad16422d8fef9a2153c2b173 P 941d3d5f8da64b18ac96345644de3869 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce330baa60b343afac700000a3f99c82 in term 1.
I20260925 19:04:55.644840 2816 raft_consensus.cc:2454] T cc94e22bad16422d8fef9a2153c2b173 P 40adefa7979c40d59796453a098c6a55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce330baa60b343afac700000a3f99c82 in term 1.
I20260925 19:04:55.644934 2607 leader_election.cc:302] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 941d3d5f8da64b18ac96345644de3869, ce330baa60b343afac700000a3f99c82; no voters:
I20260925 19:04:55.645040 3032 raft_consensus.cc:2781] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:55.645193 3032 raft_consensus.cc:686] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82 [term 1 LEADER]: Becoming Leader. State: Replica: ce330baa60b343afac700000a3f99c82, State: Running, Role: LEADER
I20260925 19:04:55.645296 3032 consensus_queue.cc:237] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } }
I20260925 19:04:55.646088 2518 catalog_manager.cc:5495] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce330baa60b343afac700000a3f99c82 (127.0.196.193). New cstate: current_term: 1 leader_uuid: "ce330baa60b343afac700000a3f99c82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } health_report { overall_health: UNKNOWN } } }
W20260925 19:04:55.669737 3003 tablet.cc:2369] T cc94e22bad16422d8fef9a2153c2b173 P 941d3d5f8da64b18ac96345644de3869: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 19:04:55 dist-test-slave-rn98 krb5kdc[2474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:04:55.674789 2572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.667682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36102 (local address 127.0.196.254:37393)
0925 19:04:55.667743 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:55.667746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:55.667912 (+ 166us) server_negotiation.cc:407] Connection header received
0925 19:04:55.667964 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:55.667964 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:55.667974 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:55.668027 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:55.668548 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.669060 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.669760 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.670024 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.670092 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.672304 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:55.672305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:55.672305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:55.672311 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:55.673716 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:55.674105 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:55.674106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:55.674107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:55.674158 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:55.674322 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:55.674323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:55.674323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:55.674421 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:55.674469 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:55.674585 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 19:04:55.674717 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 19:04:55.674897 3043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.667643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:37393 (local address 127.0.0.1:36102)
0925 19:04:55.667831 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:55.667842 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:55.667918 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:55.668044 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:55.668044 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:55.668405 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:55.668456 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.668459 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.669139 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.669140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.669671 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.669672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.670105 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.670106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.670151 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.670731 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:55.670736 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:55.672189 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:55.673793 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:55.673801 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:55.673803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:55.674026 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:55.674221 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:55.674222 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:55.674222 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:55.674280 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:55.674469 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:55.674476 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:55.674520 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 19:04:55.674571 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 19:04:55.674786 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":121,"thread_start_us":113,"threads_started":1}
I20260925 19:04:55.676934 2518 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:36102:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 19:04:55.677171 2518 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:04:55.678755 2796 tablet_service.cc:1467] Processing CreateTablet for tablet 40c7219754ff48f9ae1b21e67ebc0898 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=85d76c165c5247cfb79065fb8c4355a5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 19:04:55.678740 2654 tablet_service.cc:1467] Processing CreateTablet for tablet 40c7219754ff48f9ae1b21e67ebc0898 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=85d76c165c5247cfb79065fb8c4355a5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 19:04:55.678854 2936 tablet_service.cc:1467] Processing CreateTablet for tablet 40c7219754ff48f9ae1b21e67ebc0898 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=85d76c165c5247cfb79065fb8c4355a5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 19:04:55.678931 2796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40c7219754ff48f9ae1b21e67ebc0898. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:55.678946 2654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40c7219754ff48f9ae1b21e67ebc0898. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:55.678979 2936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40c7219754ff48f9ae1b21e67ebc0898. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:55.680459 3025 tablet_bootstrap.cc:492] T 40c7219754ff48f9ae1b21e67ebc0898 P 941d3d5f8da64b18ac96345644de3869: Bootstrap starting.
I20260925 19:04:55.680459 3027 tablet_bootstrap.cc:492] T 40c7219754ff48f9ae1b21e67ebc0898 P 40adefa7979c40d59796453a098c6a55: Bootstrap starting.
I20260925 19:04:55.680475 3026 tablet_bootstrap.cc:492] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82: Bootstrap starting.
I20260925 19:04:55.680855 3026 tablet_bootstrap.cc:654] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:55.681087 3025 tablet_bootstrap.cc:654] T 40c7219754ff48f9ae1b21e67ebc0898 P 941d3d5f8da64b18ac96345644de3869: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:55.681214 3027 tablet_bootstrap.cc:654] T 40c7219754ff48f9ae1b21e67ebc0898 P 40adefa7979c40d59796453a098c6a55: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:55.681672 3026 tablet_bootstrap.cc:492] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82: No bootstrap required, opened a new log
I20260925 19:04:55.681741 3026 ts_tablet_manager.cc:1397] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 19:04:55.681911 3026 raft_consensus.cc:348] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:55.681962 3027 tablet_bootstrap.cc:492] T 40c7219754ff48f9ae1b21e67ebc0898 P 40adefa7979c40d59796453a098c6a55: No bootstrap required, opened a new log
I20260925 19:04:55.681995 3026 raft_consensus.cc:374] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:55.682006 3027 ts_tablet_manager.cc:1397] T 40c7219754ff48f9ae1b21e67ebc0898 P 40adefa7979c40d59796453a098c6a55: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:04:55.682027 3026 raft_consensus.cc:729] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce330baa60b343afac700000a3f99c82, State: Initialized, Role: FOLLOWER
I20260925 19:04:55.682070 3026 consensus_queue.cc:260] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:55.682175 3026 ts_tablet_manager.cc:1428] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 19:04:55.682158 3027 raft_consensus.cc:348] T 40c7219754ff48f9ae1b21e67ebc0898 P 40adefa7979c40d59796453a098c6a55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:55.682238 3027 raft_consensus.cc:374] T 40c7219754ff48f9ae1b21e67ebc0898 P 40adefa7979c40d59796453a098c6a55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:55.682265 3027 raft_consensus.cc:729] T 40c7219754ff48f9ae1b21e67ebc0898 P 40adefa7979c40d59796453a098c6a55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40adefa7979c40d59796453a098c6a55, State: Initialized, Role: FOLLOWER
I20260925 19:04:55.682307 3027 consensus_queue.cc:260] T 40c7219754ff48f9ae1b21e67ebc0898 P 40adefa7979c40d59796453a098c6a55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:55.682312 3032 tablet_replica.cc:439] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } } }
I20260925 19:04:55.682387 3032 tablet_replica.cc:442] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:04:55.682405 3027 ts_tablet_manager.cc:1428] T 40c7219754ff48f9ae1b21e67ebc0898 P 40adefa7979c40d59796453a098c6a55: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 19:04:55.682475 3031 tablet_replica.cc:439] T 40c7219754ff48f9ae1b21e67ebc0898 P 40adefa7979c40d59796453a098c6a55: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } } }
I20260925 19:04:55.682552 3031 tablet_replica.cc:442] T 40c7219754ff48f9ae1b21e67ebc0898 P 40adefa7979c40d59796453a098c6a55: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:04:55.682814 3025 tablet_bootstrap.cc:492] T 40c7219754ff48f9ae1b21e67ebc0898 P 941d3d5f8da64b18ac96345644de3869: No bootstrap required, opened a new log
I20260925 19:04:55.682863 3025 ts_tablet_manager.cc:1397] T 40c7219754ff48f9ae1b21e67ebc0898 P 941d3d5f8da64b18ac96345644de3869: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:04:55.683030 3025 raft_consensus.cc:348] T 40c7219754ff48f9ae1b21e67ebc0898 P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:55.683097 3025 raft_consensus.cc:374] T 40c7219754ff48f9ae1b21e67ebc0898 P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:55.683111 3025 raft_consensus.cc:729] T 40c7219754ff48f9ae1b21e67ebc0898 P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 941d3d5f8da64b18ac96345644de3869, State: Initialized, Role: FOLLOWER
I20260925 19:04:55.683168 3025 consensus_queue.cc:260] T 40c7219754ff48f9ae1b21e67ebc0898 P 941d3d5f8da64b18ac96345644de3869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:55.683274 3025 ts_tablet_manager.cc:1428] T 40c7219754ff48f9ae1b21e67ebc0898 P 941d3d5f8da64b18ac96345644de3869: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:55.683351 3033 tablet_replica.cc:439] T 40c7219754ff48f9ae1b21e67ebc0898 P 941d3d5f8da64b18ac96345644de3869: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } } }
I20260925 19:04:55.683468 3033 tablet_replica.cc:442] T 40c7219754ff48f9ae1b21e67ebc0898 P 941d3d5f8da64b18ac96345644de3869: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:04:55.742897 3032 raft_consensus.cc:482] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:04:55.743031 3032 raft_consensus.cc:504] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:55.743274 3032 leader_election.cc:288] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40adefa7979c40d59796453a098c6a55 (127.0.196.194:44319), 941d3d5f8da64b18ac96345644de3869 (127.0.196.195:33983)
I20260925 19:04:55.743497 2816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c7219754ff48f9ae1b21e67ebc0898" candidate_uuid: "ce330baa60b343afac700000a3f99c82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40adefa7979c40d59796453a098c6a55" is_pre_election: true
I20260925 19:04:55.743592 2816 raft_consensus.cc:2454] T 40c7219754ff48f9ae1b21e67ebc0898 P 40adefa7979c40d59796453a098c6a55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce330baa60b343afac700000a3f99c82 in term 0.
I20260925 19:04:55.743533 2956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c7219754ff48f9ae1b21e67ebc0898" candidate_uuid: "ce330baa60b343afac700000a3f99c82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "941d3d5f8da64b18ac96345644de3869" is_pre_election: true
I20260925 19:04:55.743674 2956 raft_consensus.cc:2454] T 40c7219754ff48f9ae1b21e67ebc0898 P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce330baa60b343afac700000a3f99c82 in term 0.
I20260925 19:04:55.743762 2610 leader_election.cc:302] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40adefa7979c40d59796453a098c6a55, ce330baa60b343afac700000a3f99c82; no voters:
I20260925 19:04:55.743847 3032 raft_consensus.cc:2781] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:04:55.743882 3032 raft_consensus.cc:482] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:04:55.743911 3032 raft_consensus.cc:3037] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:55.744602 3032 raft_consensus.cc:504] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:55.744750 3032 leader_election.cc:288] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82 [CANDIDATE]: Term 1 election: Requested vote from peers 40adefa7979c40d59796453a098c6a55 (127.0.196.194:44319), 941d3d5f8da64b18ac96345644de3869 (127.0.196.195:33983)
I20260925 19:04:55.744944 2956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c7219754ff48f9ae1b21e67ebc0898" candidate_uuid: "ce330baa60b343afac700000a3f99c82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "941d3d5f8da64b18ac96345644de3869"
I20260925 19:04:55.745026 2956 raft_consensus.cc:3037] T 40c7219754ff48f9ae1b21e67ebc0898 P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:55.745011 2816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c7219754ff48f9ae1b21e67ebc0898" candidate_uuid: "ce330baa60b343afac700000a3f99c82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40adefa7979c40d59796453a098c6a55"
I20260925 19:04:55.745115 2816 raft_consensus.cc:3037] T 40c7219754ff48f9ae1b21e67ebc0898 P 40adefa7979c40d59796453a098c6a55 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:55.745682 2816 raft_consensus.cc:2454] T 40c7219754ff48f9ae1b21e67ebc0898 P 40adefa7979c40d59796453a098c6a55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce330baa60b343afac700000a3f99c82 in term 1.
I20260925 19:04:55.745682 2956 raft_consensus.cc:2454] T 40c7219754ff48f9ae1b21e67ebc0898 P 941d3d5f8da64b18ac96345644de3869 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce330baa60b343afac700000a3f99c82 in term 1.
I20260925 19:04:55.745900 2607 leader_election.cc:302] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 941d3d5f8da64b18ac96345644de3869, ce330baa60b343afac700000a3f99c82; no voters:
I20260925 19:04:55.746025 3032 raft_consensus.cc:2781] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:55.746106 3032 raft_consensus.cc:686] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82 [term 1 LEADER]: Becoming Leader. State: Replica: ce330baa60b343afac700000a3f99c82, State: Running, Role: LEADER
I20260925 19:04:55.746162 3032 consensus_queue.cc:237] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:55.746486 3036 tablet_replica.cc:439] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82: TxnStatusTablet state changed. Reason: New leader ce330baa60b343afac700000a3f99c82. Latest consensus state: current_term: 1 leader_uuid: "ce330baa60b343afac700000a3f99c82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } } }
I20260925 19:04:55.746596 3036 tablet_replica.cc:442] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82: This TxnStatusTablet replica's current role is: LEADER
I20260925 19:04:55.746826 3048 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 19:04:55.746966 3048 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 19:04:55.746922 2517 catalog_manager.cc:5495] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce330baa60b343afac700000a3f99c82 (127.0.196.193). New cstate: current_term: 1 leader_uuid: "ce330baa60b343afac700000a3f99c82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } health_report { overall_health: UNKNOWN } } }
I20260925 19:04:55.757449 3043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.754646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:45403 (local address 127.0.0.1:40262)
0925 19:04:55.754712 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:55.754726 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:55.754781 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:55.754965 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:55.754965 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:55.754966 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:04:55.755010 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.755011 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.755921 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.755922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.756490 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.756491 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.756955 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.756955 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.757001 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.757015 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:04:55.757304 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:04:55.757312 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 19:04:55.757354 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 19:04:55.757391 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 19:04:55.757467 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.754739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40262 (local address 127.0.196.193:45403)
0925 19:04:55.754804 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:55.754807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:55.754840 (+ 33us) server_negotiation.cc:407] Connection header received
0925 19:04:55.754874 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:55.754875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:55.754893 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:55.754959 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:55.755078 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.755846 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.756573 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.756878 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.756951 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.757106 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:55.757231 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:55.757322 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:55.757359 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 19:04:55.757382 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 19:04:55.759326 2816 raft_consensus.cc:1261] T 40c7219754ff48f9ae1b21e67ebc0898 P 40adefa7979c40d59796453a098c6a55 [term 1 FOLLOWER]: Refusing update from remote peer ce330baa60b343afac700000a3f99c82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:55.759343 2956 raft_consensus.cc:1261] T 40c7219754ff48f9ae1b21e67ebc0898 P 941d3d5f8da64b18ac96345644de3869 [term 1 FOLLOWER]: Refusing update from remote peer ce330baa60b343afac700000a3f99c82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:55.759660 3036 consensus_queue.cc:1035] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:55.759819 3032 consensus_queue.cc:1035] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:55.761103 3031 tablet_replica.cc:439] T 40c7219754ff48f9ae1b21e67ebc0898 P 40adefa7979c40d59796453a098c6a55: TxnStatusTablet state changed. Reason: New leader ce330baa60b343afac700000a3f99c82. Latest consensus state: current_term: 1 leader_uuid: "ce330baa60b343afac700000a3f99c82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } } }
I20260925 19:04:55.761222 3031 tablet_replica.cc:442] T 40c7219754ff48f9ae1b21e67ebc0898 P 40adefa7979c40d59796453a098c6a55: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:04:55.761533 3033 tablet_replica.cc:439] T 40c7219754ff48f9ae1b21e67ebc0898 P 941d3d5f8da64b18ac96345644de3869: TxnStatusTablet state changed. Reason: New leader ce330baa60b343afac700000a3f99c82. Latest consensus state: current_term: 1 leader_uuid: "ce330baa60b343afac700000a3f99c82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } } }
I20260925 19:04:55.761642 3033 tablet_replica.cc:442] T 40c7219754ff48f9ae1b21e67ebc0898 P 941d3d5f8da64b18ac96345644de3869: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:04:55.762890 3036 tablet_replica.cc:439] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ce330baa60b343afac700000a3f99c82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } } }
I20260925 19:04:55.762984 3036 tablet_replica.cc:442] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82: This TxnStatusTablet replica's current role is: LEADER
I20260925 19:04:55.763387 3033 tablet_replica.cc:439] T 40c7219754ff48f9ae1b21e67ebc0898 P 941d3d5f8da64b18ac96345644de3869: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ce330baa60b343afac700000a3f99c82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } } }
I20260925 19:04:55.763480 3033 tablet_replica.cc:442] T 40c7219754ff48f9ae1b21e67ebc0898 P 941d3d5f8da64b18ac96345644de3869: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:04:55.763589 3051 mvcc.cc:205] Tried to move back new op lower bound from 7333327240227643392 to 7333327240176721920. Current Snapshot: MvccSnapshot[applied={T|T < 7333327240227643392 or (T in {7333327240227643392})}]
I20260925 19:04:55.763669 3036 tablet_replica.cc:439] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ce330baa60b343afac700000a3f99c82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } } }
I20260925 19:04:55.763744 3036 tablet_replica.cc:442] T 40c7219754ff48f9ae1b21e67ebc0898 P ce330baa60b343afac700000a3f99c82: This TxnStatusTablet replica's current role is: LEADER
I20260925 19:04:55.763979 3053 mvcc.cc:205] Tried to move back new op lower bound from 7333327240227643392 to 7333327240176721920. Current Snapshot: MvccSnapshot[applied={T|T < 7333327240227643392}]
I20260925 19:04:55.764447 3031 tablet_replica.cc:439] T 40c7219754ff48f9ae1b21e67ebc0898 P 40adefa7979c40d59796453a098c6a55: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ce330baa60b343afac700000a3f99c82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } } }
I20260925 19:04:55.764539 3031 tablet_replica.cc:442] T 40c7219754ff48f9ae1b21e67ebc0898 P 40adefa7979c40d59796453a098c6a55: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:04:55.767990 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.765616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40272 (local address 127.0.196.193:45403)
0925 19:04:55.765676 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:55.765678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:55.765686 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:04:55.765728 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:55.765729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:55.765739 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:55.765794 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:55.765942 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.766435 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.767289 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.767510 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.767586 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.767740 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:55.767772 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:55.767822 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:55.767893 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 19:04:55.767912 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 19:04:55.773308 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.770394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:41177 (local address 127.0.196.193:45403)
0925 19:04:55.770433 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:55.770436 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:55.770680 (+ 244us) server_negotiation.cc:407] Connection header received
0925 19:04:55.770701 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:55.770702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:55.770711 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:55.770756 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:55.770903 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.771358 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.772558 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.772751 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.772806 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.773024 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:55.773056 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:55.773081 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:55.773170 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 19:04:55.773186 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:04:55.773772 3062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.770364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:45403 (local address 127.0.196.193:41177)
0925 19:04:55.770547 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:55.770559 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:55.770639 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:55.770781 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:55.770781 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:55.770783 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:04:55.770842 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.770843 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.771429 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.771429 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.772465 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.772468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.772830 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.772831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.772896 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.772911 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:04:55.773116 (+ 205us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:04:55.773117 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 19:04:55.773159 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 19:04:55.773198 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
I20260925 19:04:55.776130 2956 raft_consensus.cc:1261] T cc94e22bad16422d8fef9a2153c2b173 P 941d3d5f8da64b18ac96345644de3869 [term 1 FOLLOWER]: Refusing update from remote peer ce330baa60b343afac700000a3f99c82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:55.776104 2816 raft_consensus.cc:1261] T cc94e22bad16422d8fef9a2153c2b173 P 40adefa7979c40d59796453a098c6a55 [term 1 FOLLOWER]: Refusing update from remote peer ce330baa60b343afac700000a3f99c82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:55.776360 3063 consensus_queue.cc:1035] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:55.776438 3036 consensus_queue.cc:1035] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:55.788312 2518 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:04:55.788426 2518 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:04:55.789206 2518 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e404699963a401481b86db6814c2299: Sending DeleteTablet for 3 replicas of tablet cc94e22bad16422d8fef9a2153c2b173
I20260925 19:04:55.789528 2796 tablet_service.cc:1514] Processing DeleteTablet for tablet cc94e22bad16422d8fef9a2153c2b173 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:55 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:42664
I20260925 19:04:55.789532 2654 tablet_service.cc:1514] Processing DeleteTablet for tablet cc94e22bad16422d8fef9a2153c2b173 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:55 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:40256
I20260925 19:04:55.789542 2936 tablet_service.cc:1514] Processing DeleteTablet for tablet cc94e22bad16422d8fef9a2153c2b173 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:55 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:60204
I20260925 19:04:55.789772 3070 tablet_replica.cc:330] T cc94e22bad16422d8fef9a2153c2b173 P 941d3d5f8da64b18ac96345644de3869: stopping tablet replica
I20260925 19:04:55.789872 3070 raft_consensus.cc:2229] T cc94e22bad16422d8fef9a2153c2b173 P 941d3d5f8da64b18ac96345644de3869 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:04:55.789922 3070 raft_consensus.cc:2258] T cc94e22bad16422d8fef9a2153c2b173 P 941d3d5f8da64b18ac96345644de3869 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:55.790024 2518 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:04:55.790184 2518 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:04:55.790968 3070 ts_tablet_manager.cc:1905] T cc94e22bad16422d8fef9a2153c2b173 P 941d3d5f8da64b18ac96345644de3869: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:55.791411 2936 tablet_service.cc:1467] Processing CreateTablet for tablet 751e7eca563043caa569d8e070e505bb (DEFAULT_TABLE table=test-table [id=e2de564b1c5d41aea74ab6d0039c81dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:55.791409 2796 tablet_service.cc:1467] Processing CreateTablet for tablet 751e7eca563043caa569d8e070e505bb (DEFAULT_TABLE table=test-table [id=e2de564b1c5d41aea74ab6d0039c81dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:55.791428 2654 tablet_service.cc:1467] Processing CreateTablet for tablet 751e7eca563043caa569d8e070e505bb (DEFAULT_TABLE table=test-table [id=e2de564b1c5d41aea74ab6d0039c81dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:55.791533 2936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 751e7eca563043caa569d8e070e505bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:55.791533 2796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 751e7eca563043caa569d8e070e505bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:55.791538 2654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 751e7eca563043caa569d8e070e505bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:55.792855 3025 tablet_bootstrap.cc:492] T 751e7eca563043caa569d8e070e505bb P 941d3d5f8da64b18ac96345644de3869: Bootstrap starting.
I20260925 19:04:55.792853 3026 tablet_bootstrap.cc:492] T 751e7eca563043caa569d8e070e505bb P ce330baa60b343afac700000a3f99c82: Bootstrap starting.
I20260925 19:04:55.792878 3027 tablet_bootstrap.cc:492] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55: Bootstrap starting.
I20260925 19:04:55.793202 3027 tablet_bootstrap.cc:654] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:55.793267 3025 tablet_bootstrap.cc:654] T 751e7eca563043caa569d8e070e505bb P 941d3d5f8da64b18ac96345644de3869: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:55.793274 3026 tablet_bootstrap.cc:654] T 751e7eca563043caa569d8e070e505bb P ce330baa60b343afac700000a3f99c82: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:55.794104 3027 tablet_bootstrap.cc:492] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55: No bootstrap required, opened a new log
I20260925 19:04:55.794178 3027 ts_tablet_manager.cc:1397] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:04:55.794337 3027 raft_consensus.cc:348] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:55.794418 3027 raft_consensus.cc:374] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:55.794446 3027 raft_consensus.cc:729] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40adefa7979c40d59796453a098c6a55, State: Initialized, Role: FOLLOWER
I20260925 19:04:55.794485 3027 consensus_queue.cc:260] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:55.794579 3027 ts_tablet_manager.cc:1428] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 19:04:55.795112 3025 tablet_bootstrap.cc:492] T 751e7eca563043caa569d8e070e505bb P 941d3d5f8da64b18ac96345644de3869: No bootstrap required, opened a new log
I20260925 19:04:55.795163 3025 ts_tablet_manager.cc:1397] T 751e7eca563043caa569d8e070e505bb P 941d3d5f8da64b18ac96345644de3869: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:04:55.795331 3025 raft_consensus.cc:348] T 751e7eca563043caa569d8e070e505bb P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:55.795418 3025 raft_consensus.cc:374] T 751e7eca563043caa569d8e070e505bb P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:55.795439 3025 raft_consensus.cc:729] T 751e7eca563043caa569d8e070e505bb P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 941d3d5f8da64b18ac96345644de3869, State: Initialized, Role: FOLLOWER
I20260925 19:04:55.795482 3025 consensus_queue.cc:260] T 751e7eca563043caa569d8e070e505bb P 941d3d5f8da64b18ac96345644de3869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:55.795579 3025 ts_tablet_manager.cc:1428] T 751e7eca563043caa569d8e070e505bb P 941d3d5f8da64b18ac96345644de3869: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:55.795665 3070 ts_tablet_manager.cc:1918] T cc94e22bad16422d8fef9a2153c2b173 P 941d3d5f8da64b18ac96345644de3869: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:04:55.795758 3070 log.cc:1196] T cc94e22bad16422d8fef9a2153c2b173 P 941d3d5f8da64b18ac96345644de3869: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-2/wal/wals/cc94e22bad16422d8fef9a2153c2b173
I20260925 19:04:55.796042 3070 ts_tablet_manager.cc:1939] T cc94e22bad16422d8fef9a2153c2b173 P 941d3d5f8da64b18ac96345644de3869: Deleting consensus metadata
I20260925 19:04:55.796609 2503 catalog_manager.cc:4841] TS 941d3d5f8da64b18ac96345644de3869 (127.0.196.195:33983): tablet cc94e22bad16422d8fef9a2153c2b173 (table test-table [id=e2a5672b4699415f9291c67ed4c4f62f]) successfully deleted
I20260925 19:04:55.798728 3068 tablet_replica.cc:330] T cc94e22bad16422d8fef9a2153c2b173 P 40adefa7979c40d59796453a098c6a55: stopping tablet replica
I20260925 19:04:55.798885 3068 raft_consensus.cc:2229] T cc94e22bad16422d8fef9a2153c2b173 P 40adefa7979c40d59796453a098c6a55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:04:55.798954 3068 raft_consensus.cc:2258] T cc94e22bad16422d8fef9a2153c2b173 P 40adefa7979c40d59796453a098c6a55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:55.799039 3069 tablet_replica.cc:330] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82: stopping tablet replica
I20260925 19:04:55.799170 3026 tablet_bootstrap.cc:492] T 751e7eca563043caa569d8e070e505bb P ce330baa60b343afac700000a3f99c82: No bootstrap required, opened a new log
I20260925 19:04:55.799203 3069 raft_consensus.cc:2229] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:04:55.799229 3026 ts_tablet_manager.cc:1397] T 751e7eca563043caa569d8e070e505bb P ce330baa60b343afac700000a3f99c82: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260925 19:04:55.799350 3069 raft_consensus.cc:2258] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:55.799381 3026 raft_consensus.cc:348] T 751e7eca563043caa569d8e070e505bb P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:55.799470 3026 raft_consensus.cc:374] T 751e7eca563043caa569d8e070e505bb P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:55.799460 3068 ts_tablet_manager.cc:1905] T cc94e22bad16422d8fef9a2153c2b173 P 40adefa7979c40d59796453a098c6a55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:55.799492 3026 raft_consensus.cc:729] T 751e7eca563043caa569d8e070e505bb P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce330baa60b343afac700000a3f99c82, State: Initialized, Role: FOLLOWER
I20260925 19:04:55.799538 3026 consensus_queue.cc:260] T 751e7eca563043caa569d8e070e505bb P ce330baa60b343afac700000a3f99c82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:55.799652 3026 ts_tablet_manager.cc:1428] T 751e7eca563043caa569d8e070e505bb P ce330baa60b343afac700000a3f99c82: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 19:04:55.799809 3069 ts_tablet_manager.cc:1905] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:55.801226 3068 ts_tablet_manager.cc:1918] T cc94e22bad16422d8fef9a2153c2b173 P 40adefa7979c40d59796453a098c6a55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:04:55.801301 3068 log.cc:1196] T cc94e22bad16422d8fef9a2153c2b173 P 40adefa7979c40d59796453a098c6a55: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-1/wal/wals/cc94e22bad16422d8fef9a2153c2b173
I20260925 19:04:55.801494 3069 ts_tablet_manager.cc:1918] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:04:55.801563 3068 ts_tablet_manager.cc:1939] T cc94e22bad16422d8fef9a2153c2b173 P 40adefa7979c40d59796453a098c6a55: Deleting consensus metadata
I20260925 19:04:55.801578 3069 log.cc:1196] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-0/wal/wals/cc94e22bad16422d8fef9a2153c2b173
I20260925 19:04:55.801831 3069 ts_tablet_manager.cc:1939] T cc94e22bad16422d8fef9a2153c2b173 P ce330baa60b343afac700000a3f99c82: Deleting consensus metadata
I20260925 19:04:55.801994 2506 catalog_manager.cc:4841] TS 40adefa7979c40d59796453a098c6a55 (127.0.196.194:44319): tablet cc94e22bad16422d8fef9a2153c2b173 (table test-table [id=e2a5672b4699415f9291c67ed4c4f62f]) successfully deleted
I20260925 19:04:55.802345 2503 catalog_manager.cc:4841] TS ce330baa60b343afac700000a3f99c82 (127.0.196.193:45403): tablet cc94e22bad16422d8fef9a2153c2b173 (table test-table [id=e2a5672b4699415f9291c67ed4c4f62f]) successfully deleted
I20260925 19:04:55.828750 3031 raft_consensus.cc:482] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:04:55.828889 3031 raft_consensus.cc:504] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:55.829241 3031 leader_election.cc:288] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce330baa60b343afac700000a3f99c82 (127.0.196.193:45403), 941d3d5f8da64b18ac96345644de3869 (127.0.196.195:33983)
I20260925 19:04:55.832831 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.829527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:40433 (local address 127.0.196.193:45403)
0925 19:04:55.829576 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:55.829578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:55.829589 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:04:55.829689 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:55.829691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:55.829703 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:55.829736 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:55.829966 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.830730 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.832048 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.832413 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.832505 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.832517 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:55.832716 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 19:04:55.832760 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:04:55.832870 3074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.829407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:45403 (local address 127.0.196.194:40433)
0925 19:04:55.829521 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:55.829534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:55.829585 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:55.829803 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:55.829804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:55.829806 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:55.829886 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.829889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.830817 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.830818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.831922 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.831925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.832504 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.832505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.832589 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.832598 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 19:04:55.832701 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 19:04:55.832749 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":24,"thread_start_us":64,"threads_started":1}
I20260925 19:04:55.832952 3023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.829698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:47307 (local address 127.0.196.195:33983)
0925 19:04:55.829738 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:55.829740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:55.829747 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:04:55.829773 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:55.829774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:55.829783 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:55.829848 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:55.830001 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.830480 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.832030 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.832556 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.832656 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.832666 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:55.832831 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 19:04:55.832879 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:04:55.833220 3075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.829585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:33983 (local address 127.0.196.194:47307)
0925 19:04:55.829664 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:55.829673 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:55.829711 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:55.829848 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:55.829849 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:55.829850 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:55.829905 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.829906 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.830573 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.830574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.831922 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.831925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.832650 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.832651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.832734 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.832736 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:04:55.833091 (+ 355us) client_negotiation.cc:240] Negotiation successful
0925 19:04:55.833141 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":3,"thread_start_us":38,"threads_started":1}
I20260925 19:04:55.833178 2657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "751e7eca563043caa569d8e070e505bb" candidate_uuid: "40adefa7979c40d59796453a098c6a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce330baa60b343afac700000a3f99c82" is_pre_election: true
I20260925 19:04:55.833335 2657 raft_consensus.cc:2454] T 751e7eca563043caa569d8e070e505bb P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40adefa7979c40d59796453a098c6a55 in term 0.
I20260925 19:04:55.833411 2956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "751e7eca563043caa569d8e070e505bb" candidate_uuid: "40adefa7979c40d59796453a098c6a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "941d3d5f8da64b18ac96345644de3869" is_pre_election: true
I20260925 19:04:55.833503 2956 raft_consensus.cc:2454] T 751e7eca563043caa569d8e070e505bb P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40adefa7979c40d59796453a098c6a55 in term 0.
I20260925 19:04:55.833528 2747 leader_election.cc:302] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40adefa7979c40d59796453a098c6a55, ce330baa60b343afac700000a3f99c82; no voters:
I20260925 19:04:55.833715 3031 raft_consensus.cc:2781] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:04:55.833774 3031 raft_consensus.cc:482] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:04:55.833796 3031 raft_consensus.cc:3037] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:55.834815 3031 raft_consensus.cc:504] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:55.834959 3031 leader_election.cc:288] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55 [CANDIDATE]: Term 1 election: Requested vote from peers ce330baa60b343afac700000a3f99c82 (127.0.196.193:45403), 941d3d5f8da64b18ac96345644de3869 (127.0.196.195:33983)
I20260925 19:04:55.835175 2657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "751e7eca563043caa569d8e070e505bb" candidate_uuid: "40adefa7979c40d59796453a098c6a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce330baa60b343afac700000a3f99c82"
I20260925 19:04:55.835201 2956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "751e7eca563043caa569d8e070e505bb" candidate_uuid: "40adefa7979c40d59796453a098c6a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "941d3d5f8da64b18ac96345644de3869"
I20260925 19:04:55.835258 2657 raft_consensus.cc:3037] T 751e7eca563043caa569d8e070e505bb P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:55.835289 2956 raft_consensus.cc:3037] T 751e7eca563043caa569d8e070e505bb P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:55.835997 2956 raft_consensus.cc:2454] T 751e7eca563043caa569d8e070e505bb P 941d3d5f8da64b18ac96345644de3869 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40adefa7979c40d59796453a098c6a55 in term 1.
I20260925 19:04:55.836097 2657 raft_consensus.cc:2454] T 751e7eca563043caa569d8e070e505bb P ce330baa60b343afac700000a3f99c82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40adefa7979c40d59796453a098c6a55 in term 1.
I20260925 19:04:55.836198 2747 leader_election.cc:302] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40adefa7979c40d59796453a098c6a55, 941d3d5f8da64b18ac96345644de3869; no voters:
I20260925 19:04:55.836311 3031 raft_consensus.cc:2781] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:55.836479 3031 raft_consensus.cc:686] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55 [term 1 LEADER]: Becoming Leader. State: Replica: 40adefa7979c40d59796453a098c6a55, State: Running, Role: LEADER
I20260925 19:04:55.836553 3031 consensus_queue.cc:237] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:55.837337 2518 catalog_manager.cc:5495] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40adefa7979c40d59796453a098c6a55 (127.0.196.194). New cstate: current_term: 1 leader_uuid: "40adefa7979c40d59796453a098c6a55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } health_report { overall_health: UNKNOWN } } }
I20260925 19:04:55.844866 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.842367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42674 (local address 127.0.196.194:44319)
0925 19:04:55.842410 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:55.842412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:55.842421 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:04:55.842492 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:55.842493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:55.842513 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:55.842596 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:55.842750 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.843270 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.844014 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.844230 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.844304 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.844491 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:55.844610 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:55.844693 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:55.844749 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 19:04:55.844772 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:04:55.850991 3080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.847611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:45403 (local address 127.0.196.194:43343)
0925 19:04:55.847758 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:55.847768 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:55.847835 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:55.847966 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:55.847966 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:55.847968 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:04:55.848061 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.848063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.849119 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.849121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.849942 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.849944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.850432 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.850434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.850526 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.850545 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:04:55.850805 (+ 260us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:04:55.850807 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:04:55.850845 (+ 38us) client_negotiation.cc:240] Negotiation successful
0925 19:04:55.850911 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260925 19:04:55.850986 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.847697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:43343 (local address 127.0.196.193:45403)
0925 19:04:55.847747 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:55.847749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:55.847848 (+ 99us) server_negotiation.cc:407] Connection header received
0925 19:04:55.847879 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:55.847880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:55.847892 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:55.847965 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:55.848174 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.848962 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.850032 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.850316 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.850399 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.850697 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:55.850736 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:55.850799 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:55.850848 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 19:04:55.850870 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 19:04:55.853741 2657 raft_consensus.cc:1261] T 751e7eca563043caa569d8e070e505bb P ce330baa60b343afac700000a3f99c82 [term 1 FOLLOWER]: Refusing update from remote peer 40adefa7979c40d59796453a098c6a55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:55.853978 2956 raft_consensus.cc:1261] T 751e7eca563043caa569d8e070e505bb P 941d3d5f8da64b18ac96345644de3869 [term 1 FOLLOWER]: Refusing update from remote peer 40adefa7979c40d59796453a098c6a55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:55.854065 3031 consensus_queue.cc:1035] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:55.854387 3031 consensus_queue.cc:1035] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:55.855728 3052 mvcc.cc:205] Tried to move back new op lower bound from 7333327240614420480 to 7333327240547184640. Current Snapshot: MvccSnapshot[applied={T|T < 7333327240614420480}]
I20260925 19:04:55.864005 3062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.861205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:44319 (local address 127.0.196.193:43281)
0925 19:04:55.861256 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:55.861269 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:55.861322 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:55.861463 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:55.861464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:55.861465 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:04:55.861516 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.861518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.862440 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.862441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.863102 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:55.863105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.863603 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.863604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:55.863675 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.863683 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:04:55.863866 (+ 183us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:04:55.863869 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:04:55.863910 (+ 41us) client_negotiation.cc:240] Negotiation successful
0925 19:04:55.863950 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 19:04:55.864015 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:55.861230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:43281 (local address 127.0.196.194:44319)
0925 19:04:55.861269 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:55.861271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:55.861327 (+ 56us) server_negotiation.cc:407] Connection header received
0925 19:04:55.861385 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:55.861386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:55.861399 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:55.861461 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:55.861572 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.862347 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.863192 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:55.863491 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:55.863566 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:55.863765 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:55.863799 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:55.863849 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:55.863916 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 19:04:55.863938 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 19:04:55.864392 3052 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 19:04:57.389987 2517 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:04:57.390096 2517 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:04:57.390894 2517 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e404699963a401481b86db6814c2299: Sending DeleteTablet for 3 replicas of tablet 751e7eca563043caa569d8e070e505bb
I20260925 19:04:57.391172 2936 tablet_service.cc:1514] Processing DeleteTablet for tablet 751e7eca563043caa569d8e070e505bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:57 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:60204
I20260925 19:04:57.391192 2796 tablet_service.cc:1514] Processing DeleteTablet for tablet 751e7eca563043caa569d8e070e505bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:57 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:42664
I20260925 19:04:57.391243 2654 tablet_service.cc:1514] Processing DeleteTablet for tablet 751e7eca563043caa569d8e070e505bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:57 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:40256
I20260925 19:04:57.391475 3106 tablet_replica.cc:330] T 751e7eca563043caa569d8e070e505bb P 941d3d5f8da64b18ac96345644de3869: stopping tablet replica
I20260925 19:04:57.391547 3106 raft_consensus.cc:2229] T 751e7eca563043caa569d8e070e505bb P 941d3d5f8da64b18ac96345644de3869 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:04:57.391578 3106 raft_consensus.cc:2258] T 751e7eca563043caa569d8e070e505bb P 941d3d5f8da64b18ac96345644de3869 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:57.391716 3107 tablet_replica.cc:330] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55: stopping tablet replica
I20260925 19:04:57.391806 3107 raft_consensus.cc:2229] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:04:57.391929 3107 raft_consensus.cc:2258] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:57.392204 3106 ts_tablet_manager.cc:1905] T 751e7eca563043caa569d8e070e505bb P 941d3d5f8da64b18ac96345644de3869: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:57.393373 3107 ts_tablet_manager.cc:1905] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:57.394071 3108 tablet_replica.cc:330] T 751e7eca563043caa569d8e070e505bb P ce330baa60b343afac700000a3f99c82: stopping tablet replica
I20260925 19:04:57.394205 3106 ts_tablet_manager.cc:1918] T 751e7eca563043caa569d8e070e505bb P 941d3d5f8da64b18ac96345644de3869: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 19:04:57.394222 3108 raft_consensus.cc:2229] T 751e7eca563043caa569d8e070e505bb P ce330baa60b343afac700000a3f99c82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:04:57.394279 3108 raft_consensus.cc:2258] T 751e7eca563043caa569d8e070e505bb P ce330baa60b343afac700000a3f99c82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:57.394282 3106 log.cc:1196] T 751e7eca563043caa569d8e070e505bb P 941d3d5f8da64b18ac96345644de3869: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-2/wal/wals/751e7eca563043caa569d8e070e505bb
I20260925 19:04:57.394519 3108 ts_tablet_manager.cc:1905] T 751e7eca563043caa569d8e070e505bb P ce330baa60b343afac700000a3f99c82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:57.394529 3106 ts_tablet_manager.cc:1939] T 751e7eca563043caa569d8e070e505bb P 941d3d5f8da64b18ac96345644de3869: Deleting consensus metadata
I20260925 19:04:57.395128 2503 catalog_manager.cc:4841] TS 941d3d5f8da64b18ac96345644de3869 (127.0.196.195:33983): tablet 751e7eca563043caa569d8e070e505bb (table test-table [id=e2de564b1c5d41aea74ab6d0039c81dd]) successfully deleted
I20260925 19:04:57.395221 3107 ts_tablet_manager.cc:1918] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 19:04:57.395298 3107 log.cc:1196] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-1/wal/wals/751e7eca563043caa569d8e070e505bb
I20260925 19:04:57.395527 3107 ts_tablet_manager.cc:1939] T 751e7eca563043caa569d8e070e505bb P 40adefa7979c40d59796453a098c6a55: Deleting consensus metadata
I20260925 19:04:57.395830 2506 catalog_manager.cc:4841] TS 40adefa7979c40d59796453a098c6a55 (127.0.196.194:44319): tablet 751e7eca563043caa569d8e070e505bb (table test-table [id=e2de564b1c5d41aea74ab6d0039c81dd]) successfully deleted
I20260925 19:04:57.396528 3108 ts_tablet_manager.cc:1918] T 751e7eca563043caa569d8e070e505bb P ce330baa60b343afac700000a3f99c82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 19:04:57.396596 3108 log.cc:1196] T 751e7eca563043caa569d8e070e505bb P ce330baa60b343afac700000a3f99c82: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-0/wal/wals/751e7eca563043caa569d8e070e505bb
I20260925 19:04:57.396814 3108 ts_tablet_manager.cc:1939] T 751e7eca563043caa569d8e070e505bb P ce330baa60b343afac700000a3f99c82: Deleting consensus metadata
I20260925 19:04:57.397184 2503 catalog_manager.cc:4841] TS ce330baa60b343afac700000a3f99c82 (127.0.196.193:45403): tablet 751e7eca563043caa569d8e070e505bb (table test-table [id=e2de564b1c5d41aea74ab6d0039c81dd]) successfully deleted
Sep 25 19:04:57 dist-test-slave-rn98 krb5kdc[2474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363097, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 19:04:57 dist-test-slave-rn98 krb5kdc[2474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363097, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:04:57.420931 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:57.408710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36110 (local address 127.0.196.254:37393)
0925 19:04:57.408842 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:57.408844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:57.408853 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:04:57.408909 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:57.408909 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:57.408920 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:57.408981 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:57.409608 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:57.410135 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:57.411123 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:57.411329 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:57.411387 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:57.417873 (+ 6486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:57.417875 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:57.417875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:57.417883 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:57.419576 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:57.419997 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:57.419998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:57.419999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:57.420056 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:57.420357 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:57.420358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:57.420358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:57.420531 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:57.420665 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:57.420758 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 19:04:57.420826 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260925 19:04:57.422711 2517 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:04:57.422885 2517 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:04:57.424347 2654 tablet_service.cc:1467] Processing CreateTablet for tablet ca229b02ccc04e9ea605a7b412fbbfa5 (DEFAULT_TABLE table=test-table [id=3d8d7c0198284f979a1a644080cfbf69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:57.424373 2936 tablet_service.cc:1467] Processing CreateTablet for tablet ca229b02ccc04e9ea605a7b412fbbfa5 (DEFAULT_TABLE table=test-table [id=3d8d7c0198284f979a1a644080cfbf69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:57.424396 2796 tablet_service.cc:1467] Processing CreateTablet for tablet ca229b02ccc04e9ea605a7b412fbbfa5 (DEFAULT_TABLE table=test-table [id=3d8d7c0198284f979a1a644080cfbf69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:57.424485 2936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca229b02ccc04e9ea605a7b412fbbfa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:57.424486 2654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca229b02ccc04e9ea605a7b412fbbfa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:57.424512 2796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca229b02ccc04e9ea605a7b412fbbfa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:57.425858 3118 tablet_bootstrap.cc:492] T ca229b02ccc04e9ea605a7b412fbbfa5 P ce330baa60b343afac700000a3f99c82: Bootstrap starting.
I20260925 19:04:57.425956 3117 tablet_bootstrap.cc:492] T ca229b02ccc04e9ea605a7b412fbbfa5 P 40adefa7979c40d59796453a098c6a55: Bootstrap starting.
I20260925 19:04:57.426204 3119 tablet_bootstrap.cc:492] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869: Bootstrap starting.
I20260925 19:04:57.426229 3118 tablet_bootstrap.cc:654] T ca229b02ccc04e9ea605a7b412fbbfa5 P ce330baa60b343afac700000a3f99c82: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:57.426360 3117 tablet_bootstrap.cc:654] T ca229b02ccc04e9ea605a7b412fbbfa5 P 40adefa7979c40d59796453a098c6a55: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:57.426672 3119 tablet_bootstrap.cc:654] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:57.426836 3118 tablet_bootstrap.cc:492] T ca229b02ccc04e9ea605a7b412fbbfa5 P ce330baa60b343afac700000a3f99c82: No bootstrap required, opened a new log
I20260925 19:04:57.426896 3118 ts_tablet_manager.cc:1397] T ca229b02ccc04e9ea605a7b412fbbfa5 P ce330baa60b343afac700000a3f99c82: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:04:57.427042 3117 tablet_bootstrap.cc:492] T ca229b02ccc04e9ea605a7b412fbbfa5 P 40adefa7979c40d59796453a098c6a55: No bootstrap required, opened a new log
I20260925 19:04:57.427055 3118 raft_consensus.cc:348] T ca229b02ccc04e9ea605a7b412fbbfa5 P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:57.427124 3117 ts_tablet_manager.cc:1397] T ca229b02ccc04e9ea605a7b412fbbfa5 P 40adefa7979c40d59796453a098c6a55: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 19:04:57.427172 3118 raft_consensus.cc:374] T ca229b02ccc04e9ea605a7b412fbbfa5 P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:57.427206 3118 raft_consensus.cc:729] T ca229b02ccc04e9ea605a7b412fbbfa5 P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce330baa60b343afac700000a3f99c82, State: Initialized, Role: FOLLOWER
I20260925 19:04:57.427300 3118 consensus_queue.cc:260] T ca229b02ccc04e9ea605a7b412fbbfa5 P ce330baa60b343afac700000a3f99c82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:57.427270 3117 raft_consensus.cc:348] T ca229b02ccc04e9ea605a7b412fbbfa5 P 40adefa7979c40d59796453a098c6a55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:57.427364 3119 tablet_bootstrap.cc:492] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869: No bootstrap required, opened a new log
I20260925 19:04:57.427377 3117 raft_consensus.cc:374] T ca229b02ccc04e9ea605a7b412fbbfa5 P 40adefa7979c40d59796453a098c6a55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:57.427400 3117 raft_consensus.cc:729] T ca229b02ccc04e9ea605a7b412fbbfa5 P 40adefa7979c40d59796453a098c6a55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40adefa7979c40d59796453a098c6a55, State: Initialized, Role: FOLLOWER
I20260925 19:04:57.427405 3119 ts_tablet_manager.cc:1397] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:04:57.427551 3119 raft_consensus.cc:348] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:57.427513 3117 consensus_queue.cc:260] T ca229b02ccc04e9ea605a7b412fbbfa5 P 40adefa7979c40d59796453a098c6a55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:57.427641 3118 ts_tablet_manager.cc:1428] T ca229b02ccc04e9ea605a7b412fbbfa5 P ce330baa60b343afac700000a3f99c82: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 19:04:57.427698 3119 raft_consensus.cc:374] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:57.427726 3119 raft_consensus.cc:729] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 941d3d5f8da64b18ac96345644de3869, State: Initialized, Role: FOLLOWER
I20260925 19:04:57.427731 3117 ts_tablet_manager.cc:1428] T ca229b02ccc04e9ea605a7b412fbbfa5 P 40adefa7979c40d59796453a098c6a55: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 19:04:57.427810 3119 consensus_queue.cc:260] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:57.428071 3119 ts_tablet_manager.cc:1428] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:04:57.431063 3123 raft_consensus.cc:482] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:04:57.431125 3123 raft_consensus.cc:504] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:57.431391 3123 leader_election.cc:288] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce330baa60b343afac700000a3f99c82 (127.0.196.193:45403), 40adefa7979c40d59796453a098c6a55 (127.0.196.194:44319)
I20260925 19:04:57.434883 3127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:57.431695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:39347 (local address 127.0.196.194:44319)
0925 19:04:57.431821 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:57.431823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:57.431830 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:04:57.431859 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:57.431859 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:57.431868 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:57.431963 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:57.432143 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:57.432903 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:57.434107 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:57.434509 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:57.434608 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:57.434684 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:57.434774 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 19:04:57.434814 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":70,"threads_started":1}
I20260925 19:04:57.434886 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:57.431549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:44319 (local address 127.0.196.195:39347)
0925 19:04:57.431711 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:57.431724 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:57.431786 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:57.431974 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:57.431974 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:57.431976 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:57.432063 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:57.432065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:57.432984 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:57.432987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:57.434012 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:57.434015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:57.434607 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:57.434608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:57.434703 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:57.434708 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:04:57.434766 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 19:04:57.434810 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":73,"thread_start_us":97,"threads_started":1}
I20260925 19:04:57.434964 3126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:57.431613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:39725 (local address 127.0.196.193:45403)
0925 19:04:57.431762 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:57.431764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:57.431913 (+ 149us) server_negotiation.cc:407] Connection header received
0925 19:04:57.431937 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:57.431938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:57.431949 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:57.432017 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:57.432130 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:57.432891 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:57.434109 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:57.434554 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:57.434712 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:57.434721 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:57.434850 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 19:04:57.434890 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":95,"threads_started":1}
I20260925 19:04:57.435086 3125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:57.431549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:45403 (local address 127.0.196.195:39725)
0925 19:04:57.431854 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:57.431861 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:57.431894 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:57.432029 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:57.432030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:57.432032 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:57.432065 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:57.432066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:57.433013 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:57.433014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:57.434013 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:57.434017 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:57.434711 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:57.434712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:57.434773 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:57.434777 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:04:57.434842 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 19:04:57.434884 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":150,"spinlock_wait_cycles":2816,"thread_start_us":94,"threads_started":1}
I20260925 19:04:57.435312 2657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca229b02ccc04e9ea605a7b412fbbfa5" candidate_uuid: "941d3d5f8da64b18ac96345644de3869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce330baa60b343afac700000a3f99c82" is_pre_election: true
I20260925 19:04:57.435421 2657 raft_consensus.cc:2454] T ca229b02ccc04e9ea605a7b412fbbfa5 P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 941d3d5f8da64b18ac96345644de3869 in term 0.
I20260925 19:04:57.435567 2816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca229b02ccc04e9ea605a7b412fbbfa5" candidate_uuid: "941d3d5f8da64b18ac96345644de3869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40adefa7979c40d59796453a098c6a55" is_pre_election: true
I20260925 19:04:57.435644 2816 raft_consensus.cc:2454] T ca229b02ccc04e9ea605a7b412fbbfa5 P 40adefa7979c40d59796453a098c6a55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 941d3d5f8da64b18ac96345644de3869 in term 0.
I20260925 19:04:57.435643 2887 leader_election.cc:302] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 941d3d5f8da64b18ac96345644de3869, ce330baa60b343afac700000a3f99c82; no voters:
I20260925 19:04:57.435788 3123 raft_consensus.cc:2781] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:04:57.435827 3123 raft_consensus.cc:482] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:04:57.435846 3123 raft_consensus.cc:3037] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:57.436612 3123 raft_consensus.cc:504] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:57.436766 3123 leader_election.cc:288] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869 [CANDIDATE]: Term 1 election: Requested vote from peers ce330baa60b343afac700000a3f99c82 (127.0.196.193:45403), 40adefa7979c40d59796453a098c6a55 (127.0.196.194:44319)
I20260925 19:04:57.436883 2657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca229b02ccc04e9ea605a7b412fbbfa5" candidate_uuid: "941d3d5f8da64b18ac96345644de3869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce330baa60b343afac700000a3f99c82"
I20260925 19:04:57.436976 2657 raft_consensus.cc:3037] T ca229b02ccc04e9ea605a7b412fbbfa5 P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:57.436913 2816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca229b02ccc04e9ea605a7b412fbbfa5" candidate_uuid: "941d3d5f8da64b18ac96345644de3869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40adefa7979c40d59796453a098c6a55"
I20260925 19:04:57.437024 2816 raft_consensus.cc:3037] T ca229b02ccc04e9ea605a7b412fbbfa5 P 40adefa7979c40d59796453a098c6a55 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:57.437603 2657 raft_consensus.cc:2454] T ca229b02ccc04e9ea605a7b412fbbfa5 P ce330baa60b343afac700000a3f99c82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 941d3d5f8da64b18ac96345644de3869 in term 1.
I20260925 19:04:57.437664 2816 raft_consensus.cc:2454] T ca229b02ccc04e9ea605a7b412fbbfa5 P 40adefa7979c40d59796453a098c6a55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 941d3d5f8da64b18ac96345644de3869 in term 1.
I20260925 19:04:57.437777 2887 leader_election.cc:302] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 941d3d5f8da64b18ac96345644de3869, ce330baa60b343afac700000a3f99c82; no voters:
I20260925 19:04:57.437911 3123 raft_consensus.cc:2781] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:57.438102 3123 raft_consensus.cc:686] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869 [term 1 LEADER]: Becoming Leader. State: Replica: 941d3d5f8da64b18ac96345644de3869, State: Running, Role: LEADER
I20260925 19:04:57.438199 3123 consensus_queue.cc:237] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:57.438930 2517 catalog_manager.cc:5495] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869 reported cstate change: term changed from 0 to 1, leader changed from <none> to 941d3d5f8da64b18ac96345644de3869 (127.0.196.195). New cstate: current_term: 1 leader_uuid: "941d3d5f8da64b18ac96345644de3869" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } health_report { overall_health: HEALTHY } } }
I20260925 19:04:57.447999 3131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:57.444589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60214 (local address 127.0.196.195:33983)
0925 19:04:57.444739 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:57.444741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:57.444750 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:04:57.444796 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:57.444797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:57.444815 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:57.444893 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:57.445063 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:57.445635 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:57.446856 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:57.447273 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:57.447357 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:57.447557 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:57.447701 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:57.447783 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:57.447821 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 19:04:57.447874 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":102,"threads_started":1}
I20260925 19:04:57.454524 3126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:57.451711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:47495 (local address 127.0.196.193:45403)
0925 19:04:57.451761 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:57.451763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:57.452040 (+ 277us) server_negotiation.cc:407] Connection header received
0925 19:04:57.452170 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:57.452171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:57.452183 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:57.452233 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:57.452388 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:57.453137 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:57.453907 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:57.454090 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:57.454144 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:57.454343 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:57.454369 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:57.454407 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:57.454442 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 19:04:57.454462 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:04:57.454541 3134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:57.451550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:45403 (local address 127.0.196.195:47495)
0925 19:04:57.451975 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:57.451987 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:57.452074 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:57.452251 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:57.452252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:57.452254 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:04:57.452323 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:57.452325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:57.453228 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:57.453229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:57.453829 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:57.453831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:57.454156 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:57.454157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:57.454257 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:57.454272 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:04:57.454407 (+ 135us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:04:57.454409 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:04:57.454439 (+ 30us) client_negotiation.cc:240] Negotiation successful
0925 19:04:57.454472 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":93,"threads_started":1}
I20260925 19:04:57.456990 2816 raft_consensus.cc:1261] T ca229b02ccc04e9ea605a7b412fbbfa5 P 40adefa7979c40d59796453a098c6a55 [term 1 FOLLOWER]: Refusing update from remote peer 941d3d5f8da64b18ac96345644de3869: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:57.456990 2657 raft_consensus.cc:1261] T ca229b02ccc04e9ea605a7b412fbbfa5 P ce330baa60b343afac700000a3f99c82 [term 1 FOLLOWER]: Refusing update from remote peer 941d3d5f8da64b18ac96345644de3869: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:57.457252 3128 consensus_queue.cc:1035] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:57.457371 3123 consensus_queue.cc:1035] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:57.465010 3131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:57.462439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:42647 (local address 127.0.196.195:33983)
0925 19:04:57.462486 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:57.462488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:57.462694 (+ 206us) server_negotiation.cc:407] Connection header received
0925 19:04:57.462729 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:57.462729 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:57.462740 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:57.462790 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:57.462937 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:57.463484 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:57.464335 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:57.464591 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:57.464700 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:57.464825 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:57.464852 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:57.464896 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:57.464931 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 19:04:57.464949 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:04:57.465023 3139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:57.462397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:33983 (local address 127.0.196.193:42647)
0925 19:04:57.462533 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:57.462561 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:57.462619 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:57.462794 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:57.462794 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:57.462796 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:04:57.462838 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:57.462839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:57.463558 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:57.463558 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:57.464213 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:57.464215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:57.464699 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:57.464700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:57.464758 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:57.464762 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:04:57.464897 (+ 135us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:04:57.464898 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 19:04:57.464928 (+ 30us) client_negotiation.cc:240] Negotiation successful
0925 19:04:57.464964 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":85,"threads_started":1}
I20260925 19:04:57.471665 2518 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:04:57.471764 2518 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:04:57.472432 2518 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e404699963a401481b86db6814c2299: Sending DeleteTablet for 3 replicas of tablet ca229b02ccc04e9ea605a7b412fbbfa5
I20260925 19:04:57.472703 2796 tablet_service.cc:1514] Processing DeleteTablet for tablet ca229b02ccc04e9ea605a7b412fbbfa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:57 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:42664
I20260925 19:04:57.472740 2654 tablet_service.cc:1514] Processing DeleteTablet for tablet ca229b02ccc04e9ea605a7b412fbbfa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:57 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:40256
I20260925 19:04:57.472795 3107 tablet_replica.cc:330] T ca229b02ccc04e9ea605a7b412fbbfa5 P 40adefa7979c40d59796453a098c6a55: stopping tablet replica
I20260925 19:04:57.472837 3107 raft_consensus.cc:2229] T ca229b02ccc04e9ea605a7b412fbbfa5 P 40adefa7979c40d59796453a098c6a55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:04:57.472842 3108 tablet_replica.cc:330] T ca229b02ccc04e9ea605a7b412fbbfa5 P ce330baa60b343afac700000a3f99c82: stopping tablet replica
I20260925 19:04:57.472875 3107 raft_consensus.cc:2258] T ca229b02ccc04e9ea605a7b412fbbfa5 P 40adefa7979c40d59796453a098c6a55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:57.472891 3108 raft_consensus.cc:2229] T ca229b02ccc04e9ea605a7b412fbbfa5 P ce330baa60b343afac700000a3f99c82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:04:57.472929 3108 raft_consensus.cc:2258] T ca229b02ccc04e9ea605a7b412fbbfa5 P ce330baa60b343afac700000a3f99c82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:57.472981 2518 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:04:57.473172 2518 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:04:57.473276 3108 ts_tablet_manager.cc:1905] T ca229b02ccc04e9ea605a7b412fbbfa5 P ce330baa60b343afac700000a3f99c82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:57.473513 3107 ts_tablet_manager.cc:1905] T ca229b02ccc04e9ea605a7b412fbbfa5 P 40adefa7979c40d59796453a098c6a55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:57.474308 2796 tablet_service.cc:1467] Processing CreateTablet for tablet 5ecd93c75db542b1b37daa8f239f5704 (DEFAULT_TABLE table=test-table [id=586afaa891364ec6a38b153055b1d66a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:57.474457 2796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ecd93c75db542b1b37daa8f239f5704. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:57.474571 2936 tablet_service.cc:1514] Processing DeleteTablet for tablet ca229b02ccc04e9ea605a7b412fbbfa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:57 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:60204
I20260925 19:04:57.474674 3106 tablet_replica.cc:330] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869: stopping tablet replica
I20260925 19:04:57.474735 3106 raft_consensus.cc:2229] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:04:57.474833 3106 raft_consensus.cc:2258] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:57.475087 3108 ts_tablet_manager.cc:1918] T ca229b02ccc04e9ea605a7b412fbbfa5 P ce330baa60b343afac700000a3f99c82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:04:57.475168 3108 log.cc:1196] T ca229b02ccc04e9ea605a7b412fbbfa5 P ce330baa60b343afac700000a3f99c82: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-0/wal/wals/ca229b02ccc04e9ea605a7b412fbbfa5
I20260925 19:04:57.475394 3108 ts_tablet_manager.cc:1939] T ca229b02ccc04e9ea605a7b412fbbfa5 P ce330baa60b343afac700000a3f99c82: Deleting consensus metadata
I20260925 19:04:57.475517 3106 ts_tablet_manager.cc:1905] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:57.476054 3117 tablet_bootstrap.cc:492] T 5ecd93c75db542b1b37daa8f239f5704 P 40adefa7979c40d59796453a098c6a55: Bootstrap starting.
I20260925 19:04:57.476330 2654 tablet_service.cc:1467] Processing CreateTablet for tablet 5ecd93c75db542b1b37daa8f239f5704 (DEFAULT_TABLE table=test-table [id=586afaa891364ec6a38b153055b1d66a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:57.476389 3117 tablet_bootstrap.cc:654] T 5ecd93c75db542b1b37daa8f239f5704 P 40adefa7979c40d59796453a098c6a55: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:57.476440 2654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ecd93c75db542b1b37daa8f239f5704. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:57.477538 3106 ts_tablet_manager.cc:1918] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:04:57.477586 3106 log.cc:1196] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-2/wal/wals/ca229b02ccc04e9ea605a7b412fbbfa5
I20260925 19:04:57.477794 3106 ts_tablet_manager.cc:1939] T ca229b02ccc04e9ea605a7b412fbbfa5 P 941d3d5f8da64b18ac96345644de3869: Deleting consensus metadata
I20260925 19:04:57.477916 3118 tablet_bootstrap.cc:492] T 5ecd93c75db542b1b37daa8f239f5704 P ce330baa60b343afac700000a3f99c82: Bootstrap starting.
I20260925 19:04:57.477912 2503 catalog_manager.cc:4841] TS ce330baa60b343afac700000a3f99c82 (127.0.196.193:45403): tablet ca229b02ccc04e9ea605a7b412fbbfa5 (table test-table [id=3d8d7c0198284f979a1a644080cfbf69]) successfully deleted
I20260925 19:04:57.478257 3118 tablet_bootstrap.cc:654] T 5ecd93c75db542b1b37daa8f239f5704 P ce330baa60b343afac700000a3f99c82: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:57.478933 3107 ts_tablet_manager.cc:1918] T ca229b02ccc04e9ea605a7b412fbbfa5 P 40adefa7979c40d59796453a098c6a55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:04:57.479004 3107 log.cc:1196] T ca229b02ccc04e9ea605a7b412fbbfa5 P 40adefa7979c40d59796453a098c6a55: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-1/wal/wals/ca229b02ccc04e9ea605a7b412fbbfa5
I20260925 19:04:57.479029 3117 tablet_bootstrap.cc:492] T 5ecd93c75db542b1b37daa8f239f5704 P 40adefa7979c40d59796453a098c6a55: No bootstrap required, opened a new log
I20260925 19:04:57.479079 3117 ts_tablet_manager.cc:1397] T 5ecd93c75db542b1b37daa8f239f5704 P 40adefa7979c40d59796453a098c6a55: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260925 19:04:57.479242 3107 ts_tablet_manager.cc:1939] T ca229b02ccc04e9ea605a7b412fbbfa5 P 40adefa7979c40d59796453a098c6a55: Deleting consensus metadata
I20260925 19:04:57.479204 3117 raft_consensus.cc:348] T 5ecd93c75db542b1b37daa8f239f5704 P 40adefa7979c40d59796453a098c6a55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:57.479282 3117 raft_consensus.cc:374] T 5ecd93c75db542b1b37daa8f239f5704 P 40adefa7979c40d59796453a098c6a55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:57.479308 3117 raft_consensus.cc:729] T 5ecd93c75db542b1b37daa8f239f5704 P 40adefa7979c40d59796453a098c6a55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40adefa7979c40d59796453a098c6a55, State: Initialized, Role: FOLLOWER
I20260925 19:04:57.479364 3117 consensus_queue.cc:260] T 5ecd93c75db542b1b37daa8f239f5704 P 40adefa7979c40d59796453a098c6a55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:57.479384 2936 tablet_service.cc:1467] Processing CreateTablet for tablet 5ecd93c75db542b1b37daa8f239f5704 (DEFAULT_TABLE table=test-table [id=586afaa891364ec6a38b153055b1d66a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:57.479468 3117 ts_tablet_manager.cc:1428] T 5ecd93c75db542b1b37daa8f239f5704 P 40adefa7979c40d59796453a098c6a55: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:57.479489 2936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ecd93c75db542b1b37daa8f239f5704. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:57.479655 2506 catalog_manager.cc:4841] TS 40adefa7979c40d59796453a098c6a55 (127.0.196.194:44319): tablet ca229b02ccc04e9ea605a7b412fbbfa5 (table test-table [id=3d8d7c0198284f979a1a644080cfbf69]) successfully deleted
I20260925 19:04:57.480585 3119 tablet_bootstrap.cc:492] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869: Bootstrap starting.
I20260925 19:04:57.480762 2503 catalog_manager.cc:4841] TS 941d3d5f8da64b18ac96345644de3869 (127.0.196.195:33983): tablet ca229b02ccc04e9ea605a7b412fbbfa5 (table test-table [id=3d8d7c0198284f979a1a644080cfbf69]) successfully deleted
I20260925 19:04:57.481004 3119 tablet_bootstrap.cc:654] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:57.481146 3118 tablet_bootstrap.cc:492] T 5ecd93c75db542b1b37daa8f239f5704 P ce330baa60b343afac700000a3f99c82: No bootstrap required, opened a new log
I20260925 19:04:57.481192 3118 ts_tablet_manager.cc:1397] T 5ecd93c75db542b1b37daa8f239f5704 P ce330baa60b343afac700000a3f99c82: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 19:04:57.481304 3118 raft_consensus.cc:348] T 5ecd93c75db542b1b37daa8f239f5704 P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:57.481382 3118 raft_consensus.cc:374] T 5ecd93c75db542b1b37daa8f239f5704 P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:57.481410 3118 raft_consensus.cc:729] T 5ecd93c75db542b1b37daa8f239f5704 P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce330baa60b343afac700000a3f99c82, State: Initialized, Role: FOLLOWER
I20260925 19:04:57.481451 3118 consensus_queue.cc:260] T 5ecd93c75db542b1b37daa8f239f5704 P ce330baa60b343afac700000a3f99c82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:57.481544 3118 ts_tablet_manager.cc:1428] T 5ecd93c75db542b1b37daa8f239f5704 P ce330baa60b343afac700000a3f99c82: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 19:04:57.481598 3119 tablet_bootstrap.cc:492] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869: No bootstrap required, opened a new log
I20260925 19:04:57.481650 3119 ts_tablet_manager.cc:1397] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:04:57.481837 3119 raft_consensus.cc:348] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:57.481927 3119 raft_consensus.cc:374] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:57.481950 3119 raft_consensus.cc:729] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 941d3d5f8da64b18ac96345644de3869, State: Initialized, Role: FOLLOWER
I20260925 19:04:57.481997 3119 consensus_queue.cc:260] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:57.482138 3119 ts_tablet_manager.cc:1428] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:57.535548 3123 raft_consensus.cc:482] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:04:57.535666 3123 raft_consensus.cc:504] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:57.535835 3123 leader_election.cc:288] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce330baa60b343afac700000a3f99c82 (127.0.196.193:45403), 40adefa7979c40d59796453a098c6a55 (127.0.196.194:44319)
I20260925 19:04:57.536046 2657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ecd93c75db542b1b37daa8f239f5704" candidate_uuid: "941d3d5f8da64b18ac96345644de3869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce330baa60b343afac700000a3f99c82" is_pre_election: true
I20260925 19:04:57.536125 2815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ecd93c75db542b1b37daa8f239f5704" candidate_uuid: "941d3d5f8da64b18ac96345644de3869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40adefa7979c40d59796453a098c6a55" is_pre_election: true
I20260925 19:04:57.536276 2657 raft_consensus.cc:2454] T 5ecd93c75db542b1b37daa8f239f5704 P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 941d3d5f8da64b18ac96345644de3869 in term 0.
I20260925 19:04:57.536295 2815 raft_consensus.cc:2454] T 5ecd93c75db542b1b37daa8f239f5704 P 40adefa7979c40d59796453a098c6a55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 941d3d5f8da64b18ac96345644de3869 in term 0.
I20260925 19:04:57.536469 2887 leader_election.cc:302] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 941d3d5f8da64b18ac96345644de3869, ce330baa60b343afac700000a3f99c82; no voters:
I20260925 19:04:57.536584 3123 raft_consensus.cc:2781] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:04:57.536653 3123 raft_consensus.cc:482] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:04:57.536682 3123 raft_consensus.cc:3037] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:57.537420 3123 raft_consensus.cc:504] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:57.537570 3123 leader_election.cc:288] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869 [CANDIDATE]: Term 1 election: Requested vote from peers ce330baa60b343afac700000a3f99c82 (127.0.196.193:45403), 40adefa7979c40d59796453a098c6a55 (127.0.196.194:44319)
I20260925 19:04:57.537729 2815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ecd93c75db542b1b37daa8f239f5704" candidate_uuid: "941d3d5f8da64b18ac96345644de3869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40adefa7979c40d59796453a098c6a55"
I20260925 19:04:57.537729 2657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ecd93c75db542b1b37daa8f239f5704" candidate_uuid: "941d3d5f8da64b18ac96345644de3869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce330baa60b343afac700000a3f99c82"
I20260925 19:04:57.537803 2815 raft_consensus.cc:3037] T 5ecd93c75db542b1b37daa8f239f5704 P 40adefa7979c40d59796453a098c6a55 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:57.537804 2657 raft_consensus.cc:3037] T 5ecd93c75db542b1b37daa8f239f5704 P ce330baa60b343afac700000a3f99c82 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:57.538558 2657 raft_consensus.cc:2454] T 5ecd93c75db542b1b37daa8f239f5704 P ce330baa60b343afac700000a3f99c82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 941d3d5f8da64b18ac96345644de3869 in term 1.
I20260925 19:04:57.538558 2815 raft_consensus.cc:2454] T 5ecd93c75db542b1b37daa8f239f5704 P 40adefa7979c40d59796453a098c6a55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 941d3d5f8da64b18ac96345644de3869 in term 1.
I20260925 19:04:57.538767 2887 leader_election.cc:302] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 941d3d5f8da64b18ac96345644de3869, ce330baa60b343afac700000a3f99c82; no voters:
I20260925 19:04:57.538864 3123 raft_consensus.cc:2781] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:57.538959 3123 raft_consensus.cc:686] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869 [term 1 LEADER]: Becoming Leader. State: Replica: 941d3d5f8da64b18ac96345644de3869, State: Running, Role: LEADER
I20260925 19:04:57.539023 3123 consensus_queue.cc:237] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } }
I20260925 19:04:57.539621 2518 catalog_manager.cc:5495] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869 reported cstate change: term changed from 0 to 1, leader changed from <none> to 941d3d5f8da64b18ac96345644de3869 (127.0.196.195). New cstate: current_term: 1 leader_uuid: "941d3d5f8da64b18ac96345644de3869" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "941d3d5f8da64b18ac96345644de3869" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33983 } health_report { overall_health: HEALTHY } } }
I20260925 19:04:57.560202 2816 raft_consensus.cc:1261] T 5ecd93c75db542b1b37daa8f239f5704 P 40adefa7979c40d59796453a098c6a55 [term 1 FOLLOWER]: Refusing update from remote peer 941d3d5f8da64b18ac96345644de3869: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:57.560199 2657 raft_consensus.cc:1261] T 5ecd93c75db542b1b37daa8f239f5704 P ce330baa60b343afac700000a3f99c82 [term 1 FOLLOWER]: Refusing update from remote peer 941d3d5f8da64b18ac96345644de3869: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:57.560444 3137 consensus_queue.cc:1035] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40adefa7979c40d59796453a098c6a55" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:57.560570 3123 consensus_queue.cc:1035] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce330baa60b343afac700000a3f99c82" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 19:04:57.565711 3088 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 19:04:59.088774 2518 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:04:59.088881 2518 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:04:59.089784 2518 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e404699963a401481b86db6814c2299: Sending DeleteTablet for 3 replicas of tablet 5ecd93c75db542b1b37daa8f239f5704
I20260925 19:04:59.090094 2796 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ecd93c75db542b1b37daa8f239f5704 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:59 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:42664
I20260925 19:04:59.090073 2652 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ecd93c75db542b1b37daa8f239f5704 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:59 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:40256
I20260925 19:04:59.090247 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 2591
I20260925 19:04:59.090345 2936 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ecd93c75db542b1b37daa8f239f5704 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:59 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:60204
I20260925 19:04:59.091331 3169 tablet_replica.cc:330] T 5ecd93c75db542b1b37daa8f239f5704 P 40adefa7979c40d59796453a098c6a55: stopping tablet replica
I20260925 19:04:59.091451 3169 raft_consensus.cc:2229] T 5ecd93c75db542b1b37daa8f239f5704 P 40adefa7979c40d59796453a098c6a55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:04:59.091631 3169 raft_consensus.cc:2258] T 5ecd93c75db542b1b37daa8f239f5704 P 40adefa7979c40d59796453a098c6a55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:59.091828 3170 tablet_replica.cc:330] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869: stopping tablet replica
I20260925 19:04:59.091929 3170 raft_consensus.cc:2229] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:04:59.091984 3169 ts_tablet_manager.cc:1905] T 5ecd93c75db542b1b37daa8f239f5704 P 40adefa7979c40d59796453a098c6a55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:59.092069 3170 raft_consensus.cc:2258] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:59.092382 3170 ts_tablet_manager.cc:1905] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:59.093966 3169 ts_tablet_manager.cc:1918] T 5ecd93c75db542b1b37daa8f239f5704 P 40adefa7979c40d59796453a098c6a55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 19:04:59.094053 3169 log.cc:1196] T 5ecd93c75db542b1b37daa8f239f5704 P 40adefa7979c40d59796453a098c6a55: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-1/wal/wals/5ecd93c75db542b1b37daa8f239f5704
I20260925 19:04:59.094077 3170 ts_tablet_manager.cc:1918] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 19:04:59.094177 3170 log.cc:1196] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790363087086945-787-0/minicluster-data/ts-2/wal/wals/5ecd93c75db542b1b37daa8f239f5704
I20260925 19:04:59.094285 3169 ts_tablet_manager.cc:1939] T 5ecd93c75db542b1b37daa8f239f5704 P 40adefa7979c40d59796453a098c6a55: Deleting consensus metadata
I20260925 19:04:59.094431 3170 ts_tablet_manager.cc:1939] T 5ecd93c75db542b1b37daa8f239f5704 P 941d3d5f8da64b18ac96345644de3869: Deleting consensus metadata
I20260925 19:04:59.094699 2506 catalog_manager.cc:4841] TS 40adefa7979c40d59796453a098c6a55 (127.0.196.194:44319): tablet 5ecd93c75db542b1b37daa8f239f5704 (table test-table [id=586afaa891364ec6a38b153055b1d66a]) successfully deleted
I20260925 19:04:59.094869 2503 catalog_manager.cc:4841] TS 941d3d5f8da64b18ac96345644de3869 (127.0.196.195:33983): tablet 5ecd93c75db542b1b37daa8f239f5704 (table test-table [id=586afaa891364ec6a38b153055b1d66a]) successfully deleted
W20260925 19:04:59.096923 2503 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.196.193:45403 (error 108)
I20260925 19:04:59.097160 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 2731
I20260925 19:04:59.097952 3172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:59.097351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:45403 (local address 127.0.0.1:40280)
0925 19:04:59.097845 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:59.097899 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.193:45403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":61,"threads_started":1}
W20260925 19:04:59.098078 2503 catalog_manager.cc:4584] TS ce330baa60b343afac700000a3f99c82 (127.0.196.193:45403): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5ecd93c75db542b1b37daa8f239f5704: Network error: Client connection negotiation failed: client connection to 127.0.196.193:45403: connect: Connection refused (error 111)
I20260925 19:04:59.103888 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 2871
I20260925 19:04:59.109783 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 2487
2026-09-25T19:04:59Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5613 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:04:59 dist-test-slave-rn98 krb5kdc[3175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:04:59 dist-test-slave-rn98 krb5kdc[3175](info): set up 2 sockets
Sep 25 19:04:59 dist-test-slave-rn98 krb5kdc[3175](info): commencing operation
krb5kdc: starting...
W20260925 19:05:00.158522 787 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:05:00 dist-test-slave-rn98 krb5kdc[3175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:05:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:05:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:05:00.285713 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:35237
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:38749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:35237
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:00.368356 3188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:00.368629 3188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:00.368669 3188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:00.369884 3188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:05:00.369967 3188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:00.369990 3188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:00.370012 3188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:05:00.370043 3188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:05:00.371678 3188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:38749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:35237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:35237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:00.371995 3188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:00.372246 3188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:00.377501 3196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:00.377538 3194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:00.377683 3188 server_base.cc:1034] running on GCE node
W20260925 19:05:00.377540 3193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:00.378058 3188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:00.378456 3188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:00.379657 3188 hybrid_clock.cc:648] HybridClock initialized: now 1790363100379632 us; error 51 us; skew 500 ppm
Sep 25 19:05:00 dist-test-slave-rn98 krb5kdc[3175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:00.382455 3188 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:05:00.383019 3188 webserver.cc:460] Webserver started at http://127.0.196.254:38775/ using document root <none> and password file <none>
I20260925 19:05:00.383281 3188 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:00.383350 3188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:00.383498 3188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:00.384603 3188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "55a449861a8b473a91c8e84dbdf73158"
format_stamp: "Formatted at 2026-09-25 19:05:00 on dist-test-slave-rn98"
server_key: "2043dce85544161c0b0ebcc379defe7d"
server_key_iv: "d0cfcc90857852608bcb2191b2a21f33"
server_key_version: "encryptionkey@0"
I20260925 19:05:00.384977 3188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "55a449861a8b473a91c8e84dbdf73158"
format_stamp: "Formatted at 2026-09-25 19:05:00 on dist-test-slave-rn98"
server_key: "2043dce85544161c0b0ebcc379defe7d"
server_key_iv: "d0cfcc90857852608bcb2191b2a21f33"
server_key_version: "encryptionkey@0"
I20260925 19:05:00.386389 3188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:05:00.387284 3203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:00.387477 3188 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:05:00.387575 3188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "55a449861a8b473a91c8e84dbdf73158"
format_stamp: "Formatted at 2026-09-25 19:05:00 on dist-test-slave-rn98"
server_key: "2043dce85544161c0b0ebcc379defe7d"
server_key_iv: "d0cfcc90857852608bcb2191b2a21f33"
server_key_version: "encryptionkey@0"
I20260925 19:05:00.387665 3188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:00.406105 3188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:00.408619 3188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:00.408795 3188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:00.413028 3188 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:35237
I20260925 19:05:00.413065 3255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:35237 every 8 connection(s)
I20260925 19:05:00.413595 3188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:05:00.414295 3256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:00.416620 3256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55a449861a8b473a91c8e84dbdf73158: Bootstrap starting.
I20260925 19:05:00.417201 3256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55a449861a8b473a91c8e84dbdf73158: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:00.417443 3256 log.cc:824] T 00000000000000000000000000000000 P 55a449861a8b473a91c8e84dbdf73158: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:00.418116 3256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55a449861a8b473a91c8e84dbdf73158: No bootstrap required, opened a new log
I20260925 19:05:00.419783 3256 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55a449861a8b473a91c8e84dbdf73158 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a449861a8b473a91c8e84dbdf73158" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 35237 } }
I20260925 19:05:00.419930 3256 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55a449861a8b473a91c8e84dbdf73158 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:00.419952 3256 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55a449861a8b473a91c8e84dbdf73158 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a449861a8b473a91c8e84dbdf73158, State: Initialized, Role: FOLLOWER
I20260925 19:05:00.420059 3256 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55a449861a8b473a91c8e84dbdf73158 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a449861a8b473a91c8e84dbdf73158" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 35237 } }
I20260925 19:05:00.420146 3256 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55a449861a8b473a91c8e84dbdf73158 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:05:00.420179 3256 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55a449861a8b473a91c8e84dbdf73158 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:05:00.420235 3256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55a449861a8b473a91c8e84dbdf73158 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:00.421020 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 3188
I20260925 19:05:00.421124 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:05:00.421316 787 external_mini_cluster.cc:1428] Setting key 0a69f6c27f6e3c36212496e953f4d457
I20260925 19:05:00.421401 3256 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55a449861a8b473a91c8e84dbdf73158 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a449861a8b473a91c8e84dbdf73158" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 35237 } }
I20260925 19:05:00.421514 3256 leader_election.cc:302] T 00000000000000000000000000000000 P 55a449861a8b473a91c8e84dbdf73158 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55a449861a8b473a91c8e84dbdf73158; no voters:
I20260925 19:05:00.421665 3256 leader_election.cc:288] T 00000000000000000000000000000000 P 55a449861a8b473a91c8e84dbdf73158 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:05:00.421715 3259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55a449861a8b473a91c8e84dbdf73158 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:05:00.421960 3256 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55a449861a8b473a91c8e84dbdf73158 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:05:00.422113 3259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55a449861a8b473a91c8e84dbdf73158 [term 1 LEADER]: Becoming Leader. State: Replica: 55a449861a8b473a91c8e84dbdf73158, State: Running, Role: LEADER
I20260925 19:05:00.422338 3259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55a449861a8b473a91c8e84dbdf73158 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a449861a8b473a91c8e84dbdf73158" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 35237 } }
I20260925 19:05:00.422786 3263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55a449861a8b473a91c8e84dbdf73158 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55a449861a8b473a91c8e84dbdf73158. Latest consensus state: current_term: 1 leader_uuid: "55a449861a8b473a91c8e84dbdf73158" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a449861a8b473a91c8e84dbdf73158" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 35237 } } }
I20260925 19:05:00.422865 3263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55a449861a8b473a91c8e84dbdf73158 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:00.423151 3260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55a449861a8b473a91c8e84dbdf73158 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55a449861a8b473a91c8e84dbdf73158" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a449861a8b473a91c8e84dbdf73158" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 35237 } } }
I20260925 19:05:00.423272 3260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55a449861a8b473a91c8e84dbdf73158 [sys.catalog]: This master's current role is: LEADER
W20260925 19:05:00.423928 3277 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 55a449861a8b473a91c8e84dbdf73158: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:05:00.423987 3277 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:05:00.424033 3270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:05:00.424214 3270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:05:00.425714 3270 catalog_manager.cc:1347] Generated new cluster ID: 5ff42c7c6a204d2d9ad73f20d9d7bf3d
I20260925 19:05:00.425767 3270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:05:00 dist-test-slave-rn98 krb5kdc[3175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:05:00.433496 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:00.421935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34756 (local address 127.0.196.254:35237)
0925 19:05:00.422283 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:00.422294 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:00.422329 (+ 35us) server_negotiation.cc:407] Connection header received
0925 19:05:00.422465 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:00.422475 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:00.422545 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:00.422783 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:00.424182 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:00.425438 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:00.426328 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:00.426900 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:00.427029 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:00.429389 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:00.429394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:00.429394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:00.429408 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:00.431759 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:00.432222 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:00.432226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:00.432226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:00.432332 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:00.432551 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:00.432552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:00.432552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:00.432839 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:00.433033 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:00.433111 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 19:05:00.433259 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":76,"threads_started":1}
I20260925 19:05:00.437952 3270 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:05:00.438720 3270 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:05:00.443619 3270 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55a449861a8b473a91c8e84dbdf73158: Generated new TSK 0
I20260925 19:05:00.443823 3270 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
I20260925 19:05:00.474283 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:0
--local_ip_for_outbound_sockets=127.0.196.193
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:35237
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:38749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:00.558946 3282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:00.559178 3282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:00.559207 3282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:00.560463 3282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:00.560523 3282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:00.560552 3282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:05:00.562340 3282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:38749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:35237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:00.562626 3282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:00.562933 3282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:00.568195 3287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:00.568246 3288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:00.568241 3290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:00.568745 3282 server_base.cc:1034] running on GCE node
I20260925 19:05:00.569041 3282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:00.569317 3282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:00.570487 3282 hybrid_clock.cc:648] HybridClock initialized: now 1790363100570478 us; error 47 us; skew 500 ppm
Sep 25 19:05:00 dist-test-slave-rn98 krb5kdc[3175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:00.573184 3282 init.cc:377] Logged in from keytab as kudu/127.0.196.193@KRBTEST.COM (short username kudu)
I20260925 19:05:00.573606 3282 webserver.cc:460] Webserver started at http://127.0.196.193:43229/ using document root <none> and password file <none>
I20260925 19:05:00.573839 3282 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:00.573910 3282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:00.574066 3282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:00.575277 3282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/data/instance:
uuid: "0ce9fe7ab8fd4a66a7a7a8e80dcce85a"
format_stamp: "Formatted at 2026-09-25 19:05:00 on dist-test-slave-rn98"
server_key: "d966693ac4041f99246e108ef4c4bedb"
server_key_iv: "497b8c8839100c3080a5a041c5e8a949"
server_key_version: "encryptionkey@0"
I20260925 19:05:00.575656 3282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/wal/instance:
uuid: "0ce9fe7ab8fd4a66a7a7a8e80dcce85a"
format_stamp: "Formatted at 2026-09-25 19:05:00 on dist-test-slave-rn98"
server_key: "d966693ac4041f99246e108ef4c4bedb"
server_key_iv: "497b8c8839100c3080a5a041c5e8a949"
server_key_version: "encryptionkey@0"
I20260925 19:05:00.577061 3282 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:05:00.577932 3297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:00.578114 3282 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:00.578284 3282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "0ce9fe7ab8fd4a66a7a7a8e80dcce85a"
format_stamp: "Formatted at 2026-09-25 19:05:00 on dist-test-slave-rn98"
server_key: "d966693ac4041f99246e108ef4c4bedb"
server_key_iv: "497b8c8839100c3080a5a041c5e8a949"
server_key_version: "encryptionkey@0"
I20260925 19:05:00.578373 3282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:00.590042 3282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:00.592758 3282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:00.592939 3282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:00.593205 3282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:00.593542 3282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:00.593598 3282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:00.593643 3282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:00.593668 3282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:00.599633 3282 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:39459
I20260925 19:05:00.599730 3410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:39459 every 8 connection(s)
I20260925 19:05:00.600069 3282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
Sep 25 19:05:00 dist-test-slave-rn98 krb5kdc[3175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:05:00.609973 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 3282
I20260925 19:05:00.610073 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/wal/instance
I20260925 19:05:00.610266 787 external_mini_cluster.cc:1428] Setting key f34c4310ee2e35b30e443aa4deee94f1
I20260925 19:05:00.610595 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:00.600930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:40479 (local address 127.0.196.254:35237)
0925 19:05:00.600997 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:00.600999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:00.601230 (+ 231us) server_negotiation.cc:407] Connection header received
0925 19:05:00.602236 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:00.602238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:00.602253 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:00.602333 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:00.603066 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:00.603571 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:00.604636 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:00.604838 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:00.604906 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:00.607299 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:00.607301 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:00.607301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:00.607308 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:00.608825 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:00.609308 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:00.609309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:00.609309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:00.609365 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:00.609697 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:00.609698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:00.609698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:00.609901 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:00.609993 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:00.610381 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 19:05:00.610480 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 19:05:00.610898 3413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:00.600887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:35237 (local address 127.0.196.193:40479)
0925 19:05:00.601072 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:00.601117 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:00.602091 (+ 974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:00.602409 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:00.602413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:00.602777 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:00.602983 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:00.602988 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:00.603678 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:00.603679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:00.604538 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:00.604540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:00.604918 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:00.604919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:00.605023 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:00.605546 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:00.605556 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:00.607170 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:00.608911 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:00.608918 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:00.608920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:00.609197 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:00.609453 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:00.609454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:00.609455 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:00.609575 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:00.609990 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:00.609996 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:00.610204 (+ 208us) client_negotiation.cc:769] Sending connection context
0925 19:05:00.610352 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 19:05:00.610557 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":108,"threads_started":1}
I20260925 19:05:00.611474 3411 heartbeater.cc:347] Connected to a master server at 127.0.196.254:35237
I20260925 19:05:00.611595 3411 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:00.611806 3411 heartbeater.cc:511] Master 127.0.196.254:35237 requested a full tablet report, sending...
I20260925 19:05:00.612341 3220 ts_manager.cc:194] Registered new tserver with Master: 0ce9fe7ab8fd4a66a7a7a8e80dcce85a (127.0.196.193:39459)
I20260925 19:05:00.613317 3220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.193@KRBTEST.COM'} at 127.0.196.193:40479
WARNING: no policy specified for kudu/127.0.196.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
I20260925 19:05:00.638099 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:0
--local_ip_for_outbound_sockets=127.0.196.194
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:35237
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:38749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:00.721797 3416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:00.722038 3416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:00.722064 3416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:00.723517 3416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:00.723568 3416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:00.723596 3416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:05:00.725157 3416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:38749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:35237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:00.725426 3416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:00.725656 3416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:00.730904 3422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:00.730914 3421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:00.731112 3416 server_base.cc:1034] running on GCE node
W20260925 19:05:00.730926 3424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:00.731480 3416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:00.731722 3416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:00.732883 3416 hybrid_clock.cc:648] HybridClock initialized: now 1790363100732870 us; error 35 us; skew 500 ppm
Sep 25 19:05:00 dist-test-slave-rn98 krb5kdc[3175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:00.735580 3416 init.cc:377] Logged in from keytab as kudu/127.0.196.194@KRBTEST.COM (short username kudu)
I20260925 19:05:00.736045 3416 webserver.cc:460] Webserver started at http://127.0.196.194:39053/ using document root <none> and password file <none>
I20260925 19:05:00.736322 3416 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:00.736410 3416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:00.736584 3416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:00.737746 3416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/data/instance:
uuid: "fff4e40f19ab460aa2340fc57c0e0e78"
format_stamp: "Formatted at 2026-09-25 19:05:00 on dist-test-slave-rn98"
server_key: "a42d38b3cd3aeace1085ecb52764fbdb"
server_key_iv: "6ceb577e4e25f128b8729bc6a4a6bd06"
server_key_version: "encryptionkey@0"
I20260925 19:05:00.738121 3416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/wal/instance:
uuid: "fff4e40f19ab460aa2340fc57c0e0e78"
format_stamp: "Formatted at 2026-09-25 19:05:00 on dist-test-slave-rn98"
server_key: "a42d38b3cd3aeace1085ecb52764fbdb"
server_key_iv: "6ceb577e4e25f128b8729bc6a4a6bd06"
server_key_version: "encryptionkey@0"
I20260925 19:05:00.739607 3416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:05:00.740506 3431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:00.740725 3416 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:00.740836 3416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "fff4e40f19ab460aa2340fc57c0e0e78"
format_stamp: "Formatted at 2026-09-25 19:05:00 on dist-test-slave-rn98"
server_key: "a42d38b3cd3aeace1085ecb52764fbdb"
server_key_iv: "6ceb577e4e25f128b8729bc6a4a6bd06"
server_key_version: "encryptionkey@0"
I20260925 19:05:00.740916 3416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:00.761545 3416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:00.764145 3416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:00.764364 3416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:00.764631 3416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:00.764964 3416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:00.765019 3416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:00.765055 3416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:00.765082 3416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:00.771013 3416 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:41911
I20260925 19:05:00.771092 3544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:41911 every 8 connection(s)
I20260925 19:05:00.771432 3416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
I20260925 19:05:00.773376 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 3416
I20260925 19:05:00.773492 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/wal/instance
I20260925 19:05:00.773689 787 external_mini_cluster.cc:1428] Setting key 8e071299e710c0e43aafc69f0d4ed1f1
Sep 25 19:05:00 dist-test-slave-rn98 krb5kdc[3175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.196.195@KRBTEST.COM; defaulting to no policy
I20260925 19:05:00.783771 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:00.772253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:35665 (local address 127.0.196.254:35237)
0925 19:05:00.772344 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:00.772347 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:00.772552 (+ 205us) server_negotiation.cc:407] Connection header received
0925 19:05:00.773768 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:00.773770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:00.773785 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:00.773869 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:00.774657 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:00.775510 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:00.777141 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:00.777354 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:00.777421 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:00.780246 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:00.780248 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:00.780248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:00.780257 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:00.782040 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:00.782656 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:00.782658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:00.782658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:00.782720 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:00.783042 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:00.783043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:00.783044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:00.783175 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:00.783267 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:00.783619 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 19:05:00.783701 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 19:05:00.783934 3547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:00.772221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:35237 (local address 127.0.196.194:35665)
0925 19:05:00.772422 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:00.772461 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:00.773563 (+ 1102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:00.773970 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:00.773976 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:00.774262 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:00.774533 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:00.774539 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:00.775642 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:00.775643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:00.777039 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:00.777041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:00.777441 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:00.777443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:00.777602 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:00.778213 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:00.778224 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:00.780015 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:00.782165 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:00.782174 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:00.782177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:00.782532 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:00.782790 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:00.782791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:00.782791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:00.782960 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:00.783260 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:00.783267 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:00.783423 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 19:05:00.783563 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 19:05:00.783716 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":120,"threads_started":1}
I20260925 19:05:00.784379 3545 heartbeater.cc:347] Connected to a master server at 127.0.196.254:35237
I20260925 19:05:00.784489 3545 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:00.784648 3545 heartbeater.cc:511] Master 127.0.196.254:35237 requested a full tablet report, sending...
I20260925 19:05:00.785043 3220 ts_manager.cc:194] Registered new tserver with Master: fff4e40f19ab460aa2340fc57c0e0e78 (127.0.196.194:41911)
I20260925 19:05:00.785399 3220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.194@KRBTEST.COM'} at 127.0.196.194:35665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
I20260925 19:05:00.802470 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:0
--local_ip_for_outbound_sockets=127.0.196.195
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:35237
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:38749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:00.885723 3550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:00.885955 3550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:00.885979 3550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:00.887295 3550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:00.887338 3550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:00.887367 3550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:05:00.888952 3550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:38749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:35237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:00.889232 3550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:00.889472 3550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:00.894866 3555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:00.894855 3556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:00.894858 3550 server_base.cc:1034] running on GCE node
W20260925 19:05:00.894887 3558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:00.895328 3550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:00.895602 3550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:00.896793 3550 hybrid_clock.cc:648] HybridClock initialized: now 1790363100896791 us; error 42 us; skew 500 ppm
Sep 25 19:05:00 dist-test-slave-rn98 krb5kdc[3175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:00.899528 3550 init.cc:377] Logged in from keytab as kudu/127.0.196.195@KRBTEST.COM (short username kudu)
I20260925 19:05:00.900053 3550 webserver.cc:460] Webserver started at http://127.0.196.195:33135/ using document root <none> and password file <none>
I20260925 19:05:00.900303 3550 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:00.900379 3550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:00.900528 3550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:00.901598 3550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/data/instance:
uuid: "a41782e933ac47a5b8d09e90a70ff555"
format_stamp: "Formatted at 2026-09-25 19:05:00 on dist-test-slave-rn98"
server_key: "94545883377c83c4adea1a22035a0b80"
server_key_iv: "b34277eec5e233b1e998a2ca8e868e05"
server_key_version: "encryptionkey@0"
I20260925 19:05:00.901985 3550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/wal/instance:
uuid: "a41782e933ac47a5b8d09e90a70ff555"
format_stamp: "Formatted at 2026-09-25 19:05:00 on dist-test-slave-rn98"
server_key: "94545883377c83c4adea1a22035a0b80"
server_key_iv: "b34277eec5e233b1e998a2ca8e868e05"
server_key_version: "encryptionkey@0"
I20260925 19:05:00.903474 3550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:05:00.904273 3565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:00.904455 3550 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:05:00.904556 3550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "a41782e933ac47a5b8d09e90a70ff555"
format_stamp: "Formatted at 2026-09-25 19:05:00 on dist-test-slave-rn98"
server_key: "94545883377c83c4adea1a22035a0b80"
server_key_iv: "b34277eec5e233b1e998a2ca8e868e05"
server_key_version: "encryptionkey@0"
I20260925 19:05:00.904639 3550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:00.918022 3550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:00.920586 3550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:00.920764 3550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:00.921005 3550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:00.921319 3550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:00.921373 3550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:00.921416 3550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:00.921447 3550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:00.928172 3550 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:36065
I20260925 19:05:00.928251 3678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:36065 every 8 connection(s)
I20260925 19:05:00.928617 3550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
Sep 25 19:05:00 dist-test-slave-rn98 krb5kdc[3175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:05:00.937839 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 3550
I20260925 19:05:00.937935 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/wal/instance
I20260925 19:05:00.938102 787 external_mini_cluster.cc:1428] Setting key be7e72a91d56a9ee87c03008297021aa
I20260925 19:05:00.939057 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:00.929473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:48499 (local address 127.0.196.254:35237)
0925 19:05:00.929547 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:00.929550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:00.929792 (+ 242us) server_negotiation.cc:407] Connection header received
0925 19:05:00.930918 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:00.930920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:00.930934 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:00.931006 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:00.931715 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:00.932319 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:00.933369 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:00.933587 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:00.933650 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:00.936156 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:00.936159 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:00.936159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:00.936169 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:00.937429 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:00.937839 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:00.937841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:00.937841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:00.937902 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:00.938214 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:00.938216 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:00.938216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:00.938385 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:00.938467 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:00.938875 (+ 408us) server_negotiation.cc:299] Negotiation successful
0925 19:05:00.938964 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 19:05:00.939298 3681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:00.929441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:35237 (local address 127.0.196.195:48499)
0925 19:05:00.929652 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:00.929695 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:00.930792 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:00.931097 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:00.931101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:00.931414 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:00.931617 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:00.931621 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:00.932403 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:00.932404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:00.933269 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:00.933271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:00.933709 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:00.933711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:00.933871 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:00.934459 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:00.934469 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:00.936037 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:00.937508 (+ 1471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:00.937515 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:00.937517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:00.937759 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:00.937984 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:00.937985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:00.937985 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:00.938150 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:00.938457 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:00.938463 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:00.938692 (+ 229us) client_negotiation.cc:769] Sending connection context
0925 19:05:00.938840 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 19:05:00.939015 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":103,"threads_started":1}
I20260925 19:05:00.939797 3679 heartbeater.cc:347] Connected to a master server at 127.0.196.254:35237
I20260925 19:05:00.939898 3679 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:00.940042 3679 heartbeater.cc:511] Master 127.0.196.254:35237 requested a full tablet report, sending...
I20260925 19:05:00.940397 3220 ts_manager.cc:194] Registered new tserver with Master: a41782e933ac47a5b8d09e90a70ff555 (127.0.196.195:36065)
I20260925 19:05:00.940747 3220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.195@KRBTEST.COM'} at 127.0.196.195:48499
I20260925 19:05:00.949945 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 19:05:00.961361 3262 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:00.960339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34758 (local address 127.0.196.254:35237)
0925 19:05:00.960415 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:00.960417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:00.960517 (+ 100us) server_negotiation.cc:407] Connection header received
0925 19:05:00.960618 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:00.960619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:00.960630 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:00.960711 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:00.961302 (+ 591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34758: BlockingRecv error: recv got EOF from 127.0.0.1:34758 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 19:05:00.962518 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 3282
I20260925 19:05:00.966900 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 3416
I20260925 19:05:00.971328 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 3550
I20260925 19:05:00.975507 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 3188
2026-09-25T19:05:00Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1864 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:05:01 dist-test-slave-rn98 krb5kdc[3691](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:05:01 dist-test-slave-rn98 krb5kdc[3691](info): set up 2 sockets
Sep 25 19:05:01 dist-test-slave-rn98 krb5kdc[3691](info): commencing operation
krb5kdc: starting...
W20260925 19:05:02.026796 787 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:05:02 dist-test-slave-rn98 krb5kdc[3691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:05:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:05:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:05:02.155166 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:44235
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:45511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:44235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:02.239305 3704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:02.239611 3704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:02.239641 3704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:02.240885 3704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:05:02.240934 3704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:02.240949 3704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:02.240963 3704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:05:02.241017 3704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:05:02.242686 3704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:45511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:44235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:44235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:02.242977 3704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:02.243242 3704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:02.248466 3709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:02.248466 3710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:02.248488 3704 server_base.cc:1034] running on GCE node
W20260925 19:05:02.248466 3712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:02.249010 3704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:02.249439 3704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:02.250680 3704 hybrid_clock.cc:648] HybridClock initialized: now 1790363102250651 us; error 43 us; skew 500 ppm
Sep 25 19:05:02 dist-test-slave-rn98 krb5kdc[3691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:02.253621 3704 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:05:02.254076 3704 webserver.cc:460] Webserver started at http://127.0.196.254:38853/ using document root <none> and password file <none>
I20260925 19:05:02.254292 3704 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:02.254338 3704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:02.254437 3704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:02.255702 3704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "fb658a0775fc40b2b7ee9d6568e3e194"
format_stamp: "Formatted at 2026-09-25 19:05:02 on dist-test-slave-rn98"
server_key: "25eb818b9bdecb504e6e786fae6052dc"
server_key_iv: "faaf73c6cb518375c258ca2da9a5422e"
server_key_version: "encryptionkey@0"
I20260925 19:05:02.256038 3704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "fb658a0775fc40b2b7ee9d6568e3e194"
format_stamp: "Formatted at 2026-09-25 19:05:02 on dist-test-slave-rn98"
server_key: "25eb818b9bdecb504e6e786fae6052dc"
server_key_iv: "faaf73c6cb518375c258ca2da9a5422e"
server_key_version: "encryptionkey@0"
I20260925 19:05:02.257493 3704 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:05:02.258275 3719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:02.258437 3704 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:05:02.258534 3704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "fb658a0775fc40b2b7ee9d6568e3e194"
format_stamp: "Formatted at 2026-09-25 19:05:02 on dist-test-slave-rn98"
server_key: "25eb818b9bdecb504e6e786fae6052dc"
server_key_iv: "faaf73c6cb518375c258ca2da9a5422e"
server_key_version: "encryptionkey@0"
I20260925 19:05:02.258594 3704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:02.269951 3704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:02.272318 3704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:02.272445 3704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:02.276587 3704 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:44235
I20260925 19:05:02.276640 3771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:44235 every 8 connection(s)
I20260925 19:05:02.277000 3704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:05:02.277938 3772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:02.280223 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 3704
I20260925 19:05:02.280309 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:05:02.280495 787 external_mini_cluster.cc:1428] Setting key 0fc1aba1b1f4e17a64445245844a78f6
I20260925 19:05:02.280551 3772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb658a0775fc40b2b7ee9d6568e3e194: Bootstrap starting.
I20260925 19:05:02.281337 3772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb658a0775fc40b2b7ee9d6568e3e194: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:02.281639 3772 log.cc:824] T 00000000000000000000000000000000 P fb658a0775fc40b2b7ee9d6568e3e194: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:02.282377 3772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb658a0775fc40b2b7ee9d6568e3e194: No bootstrap required, opened a new log
I20260925 19:05:02.284273 3772 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb658a0775fc40b2b7ee9d6568e3e194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb658a0775fc40b2b7ee9d6568e3e194" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 44235 } }
I20260925 19:05:02.284449 3772 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb658a0775fc40b2b7ee9d6568e3e194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:02.284498 3772 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb658a0775fc40b2b7ee9d6568e3e194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb658a0775fc40b2b7ee9d6568e3e194, State: Initialized, Role: FOLLOWER
I20260925 19:05:02.284641 3772 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb658a0775fc40b2b7ee9d6568e3e194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb658a0775fc40b2b7ee9d6568e3e194" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 44235 } }
I20260925 19:05:02.284745 3772 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb658a0775fc40b2b7ee9d6568e3e194 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:05:02.284790 3772 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb658a0775fc40b2b7ee9d6568e3e194 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:05:02.284840 3772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb658a0775fc40b2b7ee9d6568e3e194 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:02.285548 3772 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb658a0775fc40b2b7ee9d6568e3e194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb658a0775fc40b2b7ee9d6568e3e194" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 44235 } }
I20260925 19:05:02.285665 3772 leader_election.cc:302] T 00000000000000000000000000000000 P fb658a0775fc40b2b7ee9d6568e3e194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb658a0775fc40b2b7ee9d6568e3e194; no voters:
I20260925 19:05:02.285833 3772 leader_election.cc:288] T 00000000000000000000000000000000 P fb658a0775fc40b2b7ee9d6568e3e194 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:05:02.285930 3777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb658a0775fc40b2b7ee9d6568e3e194 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:05:02.286113 3772 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb658a0775fc40b2b7ee9d6568e3e194 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:05:02.286168 3777 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb658a0775fc40b2b7ee9d6568e3e194 [term 1 LEADER]: Becoming Leader. State: Replica: fb658a0775fc40b2b7ee9d6568e3e194, State: Running, Role: LEADER
I20260925 19:05:02.286312 3777 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb658a0775fc40b2b7ee9d6568e3e194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb658a0775fc40b2b7ee9d6568e3e194" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 44235 } }
I20260925 19:05:02.286659 3777 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb658a0775fc40b2b7ee9d6568e3e194 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb658a0775fc40b2b7ee9d6568e3e194. Latest consensus state: current_term: 1 leader_uuid: "fb658a0775fc40b2b7ee9d6568e3e194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb658a0775fc40b2b7ee9d6568e3e194" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 44235 } } }
I20260925 19:05:02.286729 3777 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb658a0775fc40b2b7ee9d6568e3e194 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:02.286883 3779 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb658a0775fc40b2b7ee9d6568e3e194 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb658a0775fc40b2b7ee9d6568e3e194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb658a0775fc40b2b7ee9d6568e3e194" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 44235 } } }
I20260925 19:05:02.287151 3779 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb658a0775fc40b2b7ee9d6568e3e194 [sys.catalog]: This master's current role is: LEADER
Sep 25 19:05:02 dist-test-slave-rn98 krb5kdc[3691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
W20260925 19:05:02.288666 3793 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb658a0775fc40b2b7ee9d6568e3e194: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:05:02.288731 3793 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:05:02.288825 3781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:05:02.289027 3781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:05:02.290820 3781 catalog_manager.cc:1347] Generated new cluster ID: 1c1bb4c3b2e44af9a20ccee2b0c543e9
I20260925 19:05:02.290875 3781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:05:02.293013 3775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:02.281139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39530 (local address 127.0.196.254:44235)
0925 19:05:02.281341 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:02.281352 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:02.281388 (+ 36us) server_negotiation.cc:407] Connection header received
0925 19:05:02.281914 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:02.281922 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:02.281975 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:02.282139 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:02.282926 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:02.283814 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:02.284631 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:02.285172 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:02.285259 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:02.288412 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:02.288417 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:02.288418 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:02.288432 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:02.290474 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:02.290997 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:02.291001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:02.291001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:02.291114 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:02.291338 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:02.291339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:02.291339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:02.291779 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:02.291955 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:02.292600 (+ 645us) server_negotiation.cc:299] Negotiation successful
0925 19:05:02.292770 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":81,"threads_started":1}
I20260925 19:05:02.294122 787 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:05:02.298660 3781 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:05:02.299492 3781 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:05:02.306416 3781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb658a0775fc40b2b7ee9d6568e3e194: Generated new TSK 0
I20260925 19:05:02.306761 3781 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:05:02.311312 3775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:02.307166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39538 (local address 127.0.196.254:44235)
0925 19:05:02.307228 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:02.307230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:02.307347 (+ 117us) server_negotiation.cc:407] Connection header received
0925 19:05:02.307383 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:02.307384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:02.307397 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:02.307482 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:02.307850 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:02.308771 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:02.309556 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:02.309922 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:02.309991 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:02.310778 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:02.310780 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:02.310780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:05:02.311034 (+ 254us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:02.311056 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:05:02.311064 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:02.311137 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:02.311191 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 19:05:02.311220 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:05:02.314132 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 3704
2026-09-25T19:05:02Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1336 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:05:02 dist-test-slave-rn98 krb5kdc[3805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:05:02 dist-test-slave-rn98 krb5kdc[3805](info): set up 2 sockets
Sep 25 19:05:02 dist-test-slave-rn98 krb5kdc[3805](info): commencing operation
krb5kdc: starting...
W20260925 19:05:03.382846 787 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:05:03 dist-test-slave-rn98 krb5kdc[3805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:05:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:05:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:05:03.510288 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:43761
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:46655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:43761
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:03.592615 3818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:03.592902 3818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:03.592940 3818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:03.594215 3818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:05:03.594333 3818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:03.594357 3818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:03.594381 3818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:05:03.594410 3818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:05:03.596117 3818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:46655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:43761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:43761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:03.596455 3818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:03.596710 3818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:03.602155 3824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:03.602212 3826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:03.602488 3818 server_base.cc:1034] running on GCE node
W20260925 19:05:03.602304 3823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:03.602988 3818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:03.603392 3818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:03.604588 3818 hybrid_clock.cc:648] HybridClock initialized: now 1790363103604563 us; error 39 us; skew 500 ppm
Sep 25 19:05:03 dist-test-slave-rn98 krb5kdc[3805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:03.611423 3818 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:05:03.611946 3818 webserver.cc:460] Webserver started at http://127.0.196.254:39163/ using document root <none> and password file <none>
I20260925 19:05:03.612203 3818 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:03.612253 3818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:03.612435 3818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:03.613566 3818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "7ca9805c77b048fcba2d4d2cee4d1bb5"
format_stamp: "Formatted at 2026-09-25 19:05:03 on dist-test-slave-rn98"
server_key: "8958fef845b1baef0bad5c75ad63df45"
server_key_iv: "290e214367d71d7fb7e03ff940a4ae5c"
server_key_version: "encryptionkey@0"
I20260925 19:05:03.613952 3818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "7ca9805c77b048fcba2d4d2cee4d1bb5"
format_stamp: "Formatted at 2026-09-25 19:05:03 on dist-test-slave-rn98"
server_key: "8958fef845b1baef0bad5c75ad63df45"
server_key_iv: "290e214367d71d7fb7e03ff940a4ae5c"
server_key_version: "encryptionkey@0"
I20260925 19:05:03.615594 3818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:05:03.616515 3833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:03.616747 3818 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:03.616863 3818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "7ca9805c77b048fcba2d4d2cee4d1bb5"
format_stamp: "Formatted at 2026-09-25 19:05:03 on dist-test-slave-rn98"
server_key: "8958fef845b1baef0bad5c75ad63df45"
server_key_iv: "290e214367d71d7fb7e03ff940a4ae5c"
server_key_version: "encryptionkey@0"
I20260925 19:05:03.616945 3818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:03.626868 3818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:03.629115 3818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:03.629324 3818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:03.633649 3818 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:43761
I20260925 19:05:03.633682 3885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:43761 every 8 connection(s)
I20260925 19:05:03.634065 3818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:05:03.635260 3886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:03.635491 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 3818
I20260925 19:05:03.635604 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:05:03.635797 787 external_mini_cluster.cc:1428] Setting key a372d4d26f9b90c52187765f8749f56f
I20260925 19:05:03.639161 3886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ca9805c77b048fcba2d4d2cee4d1bb5: Bootstrap starting.
I20260925 19:05:03.639943 3886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ca9805c77b048fcba2d4d2cee4d1bb5: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:03.640273 3886 log.cc:824] T 00000000000000000000000000000000 P 7ca9805c77b048fcba2d4d2cee4d1bb5: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:03.641274 3886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ca9805c77b048fcba2d4d2cee4d1bb5: No bootstrap required, opened a new log
Sep 25 19:05:03 dist-test-slave-rn98 krb5kdc[3805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:05:03.644018 3886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ca9805c77b048fcba2d4d2cee4d1bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ca9805c77b048fcba2d4d2cee4d1bb5" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 43761 } }
I20260925 19:05:03.644276 3886 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ca9805c77b048fcba2d4d2cee4d1bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:03.644337 3886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ca9805c77b048fcba2d4d2cee4d1bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ca9805c77b048fcba2d4d2cee4d1bb5, State: Initialized, Role: FOLLOWER
I20260925 19:05:03.644490 3886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ca9805c77b048fcba2d4d2cee4d1bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ca9805c77b048fcba2d4d2cee4d1bb5" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 43761 } }
I20260925 19:05:03.644578 3886 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ca9805c77b048fcba2d4d2cee4d1bb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:05:03.644655 3886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ca9805c77b048fcba2d4d2cee4d1bb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:05:03.644719 3886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ca9805c77b048fcba2d4d2cee4d1bb5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:03.645522 3886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ca9805c77b048fcba2d4d2cee4d1bb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ca9805c77b048fcba2d4d2cee4d1bb5" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 43761 } }
I20260925 19:05:03.645673 3886 leader_election.cc:302] T 00000000000000000000000000000000 P 7ca9805c77b048fcba2d4d2cee4d1bb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ca9805c77b048fcba2d4d2cee4d1bb5; no voters:
I20260925 19:05:03.645876 3886 leader_election.cc:288] T 00000000000000000000000000000000 P 7ca9805c77b048fcba2d4d2cee4d1bb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:05:03.645912 3891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ca9805c77b048fcba2d4d2cee4d1bb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:05:03.646034 3891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ca9805c77b048fcba2d4d2cee4d1bb5 [term 1 LEADER]: Becoming Leader. State: Replica: 7ca9805c77b048fcba2d4d2cee4d1bb5, State: Running, Role: LEADER
I20260925 19:05:03.646188 3886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ca9805c77b048fcba2d4d2cee4d1bb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:05:03.646215 3891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ca9805c77b048fcba2d4d2cee4d1bb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ca9805c77b048fcba2d4d2cee4d1bb5" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 43761 } }
I20260925 19:05:03.646600 3891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ca9805c77b048fcba2d4d2cee4d1bb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ca9805c77b048fcba2d4d2cee4d1bb5. Latest consensus state: current_term: 1 leader_uuid: "7ca9805c77b048fcba2d4d2cee4d1bb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ca9805c77b048fcba2d4d2cee4d1bb5" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 43761 } } }
I20260925 19:05:03.646788 3891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ca9805c77b048fcba2d4d2cee4d1bb5 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:03.647092 3899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:05:03.647758 3899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:05:03.648496 3892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ca9805c77b048fcba2d4d2cee4d1bb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ca9805c77b048fcba2d4d2cee4d1bb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ca9805c77b048fcba2d4d2cee4d1bb5" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 43761 } } }
I20260925 19:05:03.648676 3892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ca9805c77b048fcba2d4d2cee4d1bb5 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:03.648746 3889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:03.636376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.0.196.254:43761)
0925 19:05:03.636558 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:03.636569 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:03.636608 (+ 39us) server_negotiation.cc:407] Connection header received
0925 19:05:03.637152 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:03.637162 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:03.637239 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:03.637423 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:03.637995 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:03.639603 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:03.640495 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:03.641159 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:03.641287 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:03.644242 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:03.644247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:03.644247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:03.644260 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:03.646454 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:03.646913 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:03.646917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:03.646917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:03.647020 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:03.647290 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:03.647292 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:03.647292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:03.647648 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:03.647792 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:03.648392 (+ 600us) server_negotiation.cc:299] Negotiation successful
0925 19:05:03.648539 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":69,"threads_started":1}
I20260925 19:05:03.649760 3899 catalog_manager.cc:1347] Generated new cluster ID: f13a0a5ebfa34f75ae2506b67e354a0d
I20260925 19:05:03.649816 3899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:05:03.654611 3899 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:05:03.655309 3899 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:05:03.668054 3899 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ca9805c77b048fcba2d4d2cee4d1bb5: Generated new TSK 0
I20260925 19:05:03.668561 3899 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
I20260925 19:05:03.700215 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:0
--local_ip_for_outbound_sockets=127.0.196.193
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:43761
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:46655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:03.784816 3912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:03.785036 3912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:03.785061 3912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:03.786276 3912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:03.786326 3912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:03.786355 3912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:05:03.787905 3912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:46655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:43761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:03.788179 3912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:03.788444 3912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:03.793767 3920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:03.793834 3917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:03.793926 3918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:03.793988 3912 server_base.cc:1034] running on GCE node
I20260925 19:05:03.794337 3912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:03.794606 3912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:03.795830 3912 hybrid_clock.cc:648] HybridClock initialized: now 1790363103795812 us; error 49 us; skew 500 ppm
Sep 25 19:05:03 dist-test-slave-rn98 krb5kdc[3805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:03.798454 3912 init.cc:377] Logged in from keytab as kudu/127.0.196.193@KRBTEST.COM (short username kudu)
I20260925 19:05:03.799042 3912 webserver.cc:460] Webserver started at http://127.0.196.193:38845/ using document root <none> and password file <none>
I20260925 19:05:03.799296 3912 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:03.799365 3912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:03.799513 3912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:03.800582 3912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/data/instance:
uuid: "8c0d571833544978a9083c442d3ad3db"
format_stamp: "Formatted at 2026-09-25 19:05:03 on dist-test-slave-rn98"
server_key: "eea1d9813a377fe4c8a922bea6568e28"
server_key_iv: "29d34ce85a94d7524ac3b9ff918e9fac"
server_key_version: "encryptionkey@0"
I20260925 19:05:03.800942 3912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/wal/instance:
uuid: "8c0d571833544978a9083c442d3ad3db"
format_stamp: "Formatted at 2026-09-25 19:05:03 on dist-test-slave-rn98"
server_key: "eea1d9813a377fe4c8a922bea6568e28"
server_key_iv: "29d34ce85a94d7524ac3b9ff918e9fac"
server_key_version: "encryptionkey@0"
I20260925 19:05:03.802379 3912 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:05:03.803341 3927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:03.803571 3912 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:05:03.803682 3912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "8c0d571833544978a9083c442d3ad3db"
format_stamp: "Formatted at 2026-09-25 19:05:03 on dist-test-slave-rn98"
server_key: "eea1d9813a377fe4c8a922bea6568e28"
server_key_iv: "29d34ce85a94d7524ac3b9ff918e9fac"
server_key_version: "encryptionkey@0"
I20260925 19:05:03.803776 3912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:03.827212 3912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:03.829250 3912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:03.829433 3912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:03.829679 3912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:03.830017 3912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:03.830075 3912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:03.830129 3912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:03.830157 3912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:03.836658 3912 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:33699
I20260925 19:05:03.836700 4040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:33699 every 8 connection(s)
I20260925 19:05:03.837049 3912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
Sep 25 19:05:03 dist-test-slave-rn98 krb5kdc[3805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:05:03.846153 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 3912
I20260925 19:05:03.846326 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-0/wal/instance
I20260925 19:05:03.846609 787 external_mini_cluster.cc:1428] Setting key c48bf3ab101d55cee28308948c7ca402
I20260925 19:05:03.848232 3889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:03.837830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:58399 (local address 127.0.196.254:43761)
0925 19:05:03.837883 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:03.837885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:03.838156 (+ 271us) server_negotiation.cc:407] Connection header received
0925 19:05:03.838716 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:03.838717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:03.838728 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:03.838814 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:03.839829 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:03.840349 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:03.841515 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:03.841722 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:03.841787 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:03.844619 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:03.844627 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:03.844628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:03.844635 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:03.846273 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:03.846968 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:03.846969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:03.846970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:03.847075 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:03.847374 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:03.847375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:03.847376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:03.847538 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:03.847609 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:03.848062 (+ 453us) server_negotiation.cc:299] Negotiation successful
0925 19:05:03.848152 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:05:03.848451 4043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:03.837807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:43761 (local address 127.0.196.193:58399)
0925 19:05:03.837972 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:03.838015 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:03.838593 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:03.838877 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:03.838880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:03.839495 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:03.839708 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:03.839713 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:03.840465 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:03.840466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:03.841420 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:03.841423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:03.841836 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:03.841838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:03.842058 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:03.842855 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:03.842864 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:03.844506 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:03.846403 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:03.846412 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:03.846415 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:03.846853 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:03.847146 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:03.847147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:03.847147 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:03.847265 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:03.847620 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:03.847627 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:03.847822 (+ 195us) client_negotiation.cc:769] Sending connection context
0925 19:05:03.847942 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 19:05:03.848137 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260925 19:05:03.849184 4041 heartbeater.cc:347] Connected to a master server at 127.0.196.254:43761
I20260925 19:05:03.849306 4041 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:03.849494 4041 heartbeater.cc:511] Master 127.0.196.254:43761 requested a full tablet report, sending...
I20260925 19:05:03.850033 3850 ts_manager.cc:194] Registered new tserver with Master: 8c0d571833544978a9083c442d3ad3db (127.0.196.193:33699)
I20260925 19:05:03.850751 3850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.193@KRBTEST.COM'} at 127.0.196.193:58399
WARNING: no policy specified for kudu/127.0.196.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
I20260925 19:05:03.874054 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:0
--local_ip_for_outbound_sockets=127.0.196.194
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:43761
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:46655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:03.957531 4046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:03.957793 4046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:03.957837 4046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:03.959192 4046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:03.959267 4046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:03.959312 4046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:05:03.960810 4046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:46655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:43761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:03.961095 4046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:03.961347 4046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:03.966481 4051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:03.966467 4052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:03.966522 4054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:03.966770 4046 server_base.cc:1034] running on GCE node
I20260925 19:05:03.967100 4046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:03.967394 4046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:03.968564 4046 hybrid_clock.cc:648] HybridClock initialized: now 1790363103968537 us; error 38 us; skew 500 ppm
Sep 25 19:05:03 dist-test-slave-rn98 krb5kdc[3805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:03.971361 4046 init.cc:377] Logged in from keytab as kudu/127.0.196.194@KRBTEST.COM (short username kudu)
I20260925 19:05:03.971845 4046 webserver.cc:460] Webserver started at http://127.0.196.194:43391/ using document root <none> and password file <none>
I20260925 19:05:03.972157 4046 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:03.972237 4046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:03.972388 4046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:03.973466 4046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/data/instance:
uuid: "21d153e59d0f4014b4f69034bdfefc7f"
format_stamp: "Formatted at 2026-09-25 19:05:03 on dist-test-slave-rn98"
server_key: "1981ef74ee654855dc0440a553d689f8"
server_key_iv: "73f3db4b876497bb0e875f9ccb59a87e"
server_key_version: "encryptionkey@0"
I20260925 19:05:03.973853 4046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/wal/instance:
uuid: "21d153e59d0f4014b4f69034bdfefc7f"
format_stamp: "Formatted at 2026-09-25 19:05:03 on dist-test-slave-rn98"
server_key: "1981ef74ee654855dc0440a553d689f8"
server_key_iv: "73f3db4b876497bb0e875f9ccb59a87e"
server_key_version: "encryptionkey@0"
I20260925 19:05:03.975385 4046 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:05:03.976294 4061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:03.976475 4046 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:05:03.976585 4046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "21d153e59d0f4014b4f69034bdfefc7f"
format_stamp: "Formatted at 2026-09-25 19:05:03 on dist-test-slave-rn98"
server_key: "1981ef74ee654855dc0440a553d689f8"
server_key_iv: "73f3db4b876497bb0e875f9ccb59a87e"
server_key_version: "encryptionkey@0"
I20260925 19:05:03.976678 4046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:03.988999 4046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:03.991325 4046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:03.991504 4046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:03.991773 4046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:03.992097 4046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:03.992156 4046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:03.992239 4046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:03.992274 4046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:03.998471 4046 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:36759
I20260925 19:05:03.998548 4174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:36759 every 8 connection(s)
I20260925 19:05:03.998953 4046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
I20260925 19:05:03.999238 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 4046
I20260925 19:05:03.999370 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-1/wal/instance
I20260925 19:05:03.999596 787 external_mini_cluster.cc:1428] Setting key 33abc55ec44f627ff62e6a8f79fca3d2
Sep 25 19:05:04 dist-test-slave-rn98 krb5kdc[3805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.196.195@KRBTEST.COM; defaulting to no policy
I20260925 19:05:04.010586 3889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:03.999873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:38779 (local address 127.0.196.254:43761)
0925 19:05:03.999931 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:03.999933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:04.000195 (+ 262us) server_negotiation.cc:407] Connection header received
0925 19:05:04.000835 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:04.000836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:04.000849 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:04.000899 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:04.002391 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:04.003216 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:04.004281 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:04.004642 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:04.004746 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:04.007527 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:04.007528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:04.007529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:04.007536 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:04.009214 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:04.009643 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:04.009643 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:04.009644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:04.009693 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:04.009933 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:04.009933 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:04.009933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:04.010039 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:04.010090 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:04.010431 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 19:05:04.010516 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 19:05:04.010845 4177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:03.999808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:43761 (local address 127.0.196.194:38779)
0925 19:05:04.000022 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:04.000085 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:04.000711 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:04.000981 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:04.000986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:04.001910 (+ 924us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:04.002264 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:04.002272 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:04.003335 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:04.003336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:04.004172 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:04.004174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:04.004754 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:04.004755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:04.004864 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:04.005478 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:04.005487 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:04.007414 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:04.009302 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:04.009311 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:04.009314 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:04.009571 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:04.009766 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:04.009767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:04.009767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:04.009876 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:04.010100 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:04.010104 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:04.010297 (+ 193us) client_negotiation.cc:769] Sending connection context
0925 19:05:04.010396 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 19:05:04.010537 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":88,"threads_started":1}
I20260925 19:05:04.011435 4175 heartbeater.cc:347] Connected to a master server at 127.0.196.254:43761
I20260925 19:05:04.011549 4175 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:04.011771 4175 heartbeater.cc:511] Master 127.0.196.254:43761 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.195@KRBTEST.COM" created.
I20260925 19:05:04.012229 3850 ts_manager.cc:194] Registered new tserver with Master: 21d153e59d0f4014b4f69034bdfefc7f (127.0.196.194:36759)
I20260925 19:05:04.012621 3850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.194@KRBTEST.COM'} at 127.0.196.194:38779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
I20260925 19:05:04.030524 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:0
--local_ip_for_outbound_sockets=127.0.196.195
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:43761
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:46655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:04.115556 4180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:04.115778 4180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:04.115803 4180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:04.117127 4180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:04.117178 4180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:04.117206 4180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:05:04.118734 4180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:46655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:43761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:04.118992 4180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:04.119240 4180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:04.124629 4186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:04.124625 4185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:04.124946 4180 server_base.cc:1034] running on GCE node
W20260925 19:05:04.124660 4188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:04.125300 4180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:04.125514 4180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:04.126708 4180 hybrid_clock.cc:648] HybridClock initialized: now 1790363104126695 us; error 40 us; skew 500 ppm
Sep 25 19:05:04 dist-test-slave-rn98 krb5kdc[3805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:04.129338 4180 init.cc:377] Logged in from keytab as kudu/127.0.196.195@KRBTEST.COM (short username kudu)
I20260925 19:05:04.129870 4180 webserver.cc:460] Webserver started at http://127.0.196.195:40777/ using document root <none> and password file <none>
I20260925 19:05:04.130110 4180 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:04.130182 4180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:04.130316 4180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:04.131470 4180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/data/instance:
uuid: "e53a055b9a59429e81f44e3030b10284"
format_stamp: "Formatted at 2026-09-25 19:05:04 on dist-test-slave-rn98"
server_key: "c6bb9ec72ed6ee51617924901f074dc6"
server_key_iv: "d1f0edfe845786b17c8b4bad0249e809"
server_key_version: "encryptionkey@0"
I20260925 19:05:04.131861 4180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/wal/instance:
uuid: "e53a055b9a59429e81f44e3030b10284"
format_stamp: "Formatted at 2026-09-25 19:05:04 on dist-test-slave-rn98"
server_key: "c6bb9ec72ed6ee51617924901f074dc6"
server_key_iv: "d1f0edfe845786b17c8b4bad0249e809"
server_key_version: "encryptionkey@0"
I20260925 19:05:04.133309 4180 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 19:05:04.134163 4195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:04.134460 4180 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:05:04.134570 4180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "e53a055b9a59429e81f44e3030b10284"
format_stamp: "Formatted at 2026-09-25 19:05:04 on dist-test-slave-rn98"
server_key: "c6bb9ec72ed6ee51617924901f074dc6"
server_key_iv: "d1f0edfe845786b17c8b4bad0249e809"
server_key_version: "encryptionkey@0"
I20260925 19:05:04.134687 4180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:04.154460 4180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:04.156625 4180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:04.156801 4180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:04.157034 4180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:04.157361 4180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:04.157418 4180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:04.157459 4180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:04.157485 4180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:04.163484 4180 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:34079
I20260925 19:05:04.163579 4308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:34079 every 8 connection(s)
I20260925 19:05:04.163939 4180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
I20260925 19:05:04.166030 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 4180
I20260925 19:05:04.166131 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790363087086945-787-0/minicluster-data/ts-2/wal/instance
I20260925 19:05:04.166304 787 external_mini_cluster.cc:1428] Setting key ec91b4ed04fcc47b4b530eba352d67ec
Sep 25 19:05:04 dist-test-slave-rn98 krb5kdc[3805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:05:04.174230 3889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:04.164850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:33489 (local address 127.0.196.254:43761)
0925 19:05:04.164923 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:04.164925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:04.165137 (+ 212us) server_negotiation.cc:407] Connection header received
0925 19:05:04.165684 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:04.165685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:04.165695 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:04.165761 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:04.167181 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:04.167763 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:04.169001 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:04.169223 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:04.169282 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:04.171671 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:04.171673 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:04.171673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:04.171680 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:04.172973 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:04.173379 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:04.173380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:04.173381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:04.173429 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:04.173656 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:04.173657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:04.173657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:04.173774 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:04.173827 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:04.174083 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 19:05:04.174148 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 19:05:04.174480 4311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:04.164826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:43761 (local address 127.0.196.195:33489)
0925 19:05:04.164995 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:04.165043 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:04.165587 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:04.165828 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:04.165833 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:04.166735 (+ 902us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:04.167065 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:04.167072 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:04.167861 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:04.167862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:04.168915 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:04.168917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:04.169306 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:04.169307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:04.169422 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:04.169939 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:04.169948 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:04.171571 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:04.173051 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:04.173058 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:04.173060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:04.173311 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:04.173488 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:04.173489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:04.173489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:04.173599 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:04.173834 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:04.173839 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:04.173967 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 19:05:04.174059 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 19:05:04.174190 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260925 19:05:04.175114 4309 heartbeater.cc:347] Connected to a master server at 127.0.196.254:43761
I20260925 19:05:04.175235 4309 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:04.175436 4309 heartbeater.cc:511] Master 127.0.196.254:43761 requested a full tablet report, sending...
I20260925 19:05:04.175858 3850 ts_manager.cc:194] Registered new tserver with Master: e53a055b9a59429e81f44e3030b10284 (127.0.196.195:34079)
I20260925 19:05:04.176292 3850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.195@KRBTEST.COM'} at 127.0.196.195:33489
I20260925 19:05:04.178331 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 19:05:04 dist-test-slave-rn98 krb5kdc[3805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363104, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 19:05:04 dist-test-slave-rn98 krb5kdc[3805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363104, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:05:04.201861 3889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:04.194495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32944 (local address 127.0.196.254:43761)
0925 19:05:04.194567 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:04.194569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:04.194771 (+ 202us) server_negotiation.cc:407] Connection header received
0925 19:05:04.194830 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:04.194831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:04.194841 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:04.194895 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:04.195540 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:04.196018 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:04.196816 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:04.196989 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:04.197037 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:04.199428 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:04.199430 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:04.199430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:04.199438 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:04.200865 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:04.201196 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:04.201196 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:04.201197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:04.201245 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:04.201416 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:04.201417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:04.201417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:04.201556 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:04.201629 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:04.201737 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 19:05:04.201793 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 19:05:04.202188 3850 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:32944
I20260925 19:05:04.202744 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 3912
I20260925 19:05:04.207906 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 4046
I20260925 19:05:04.213441 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 4180
I20260925 19:05:04.217917 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 3818
2026-09-25T19:05:04Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1905 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:05:04 dist-test-slave-rn98 krb5kdc[4321](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:05:04 dist-test-slave-rn98 krb5kdc[4321](info): set up 2 sockets
Sep 25 19:05:04 dist-test-slave-rn98 krb5kdc[4321](info): commencing operation
krb5kdc: starting...
W20260925 19:05:05.274812 787 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:05:05 dist-test-slave-rn98 krb5kdc[4321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:05:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:05:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:05:05.398476 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:45225
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:35945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:45225
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:05.482574 4334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:05.482836 4334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:05.482860 4334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:05.484143 4334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:05:05.484189 4334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:05.484205 4334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:05.484217 4334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:05:05.484229 4334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:05:05.485749 4334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:35945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:45225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:45225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:05.486006 4334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:05.486282 4334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:05.491344 4339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:05.491364 4340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:05.491376 4334 server_base.cc:1034] running on GCE node
W20260925 19:05:05.491344 4342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:05.491930 4334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:05.492220 4334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:05.493368 4334 hybrid_clock.cc:648] HybridClock initialized: now 1790363105493358 us; error 32 us; skew 500 ppm
Sep 25 19:05:05 dist-test-slave-rn98 krb5kdc[4321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:05.496031 4334 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:05:05.496515 4334 webserver.cc:460] Webserver started at http://127.0.196.254:36915/ using document root <none> and password file <none>
I20260925 19:05:05.496757 4334 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:05.496825 4334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:05.496979 4334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:05.498066 4334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "3d0251be0e0d48829fd052d81d67d6b0"
format_stamp: "Formatted at 2026-09-25 19:05:05 on dist-test-slave-rn98"
server_key: "84c5bb9e55ae5b5f495c1270b189446a"
server_key_iv: "d34a11ea326025df1a0f197e650985ad"
server_key_version: "encryptionkey@0"
I20260925 19:05:05.498446 4334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "3d0251be0e0d48829fd052d81d67d6b0"
format_stamp: "Formatted at 2026-09-25 19:05:05 on dist-test-slave-rn98"
server_key: "84c5bb9e55ae5b5f495c1270b189446a"
server_key_iv: "d34a11ea326025df1a0f197e650985ad"
server_key_version: "encryptionkey@0"
I20260925 19:05:05.499858 4334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:05:05.500587 4349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:05.500763 4334 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:05.500866 4334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "3d0251be0e0d48829fd052d81d67d6b0"
format_stamp: "Formatted at 2026-09-25 19:05:05 on dist-test-slave-rn98"
server_key: "84c5bb9e55ae5b5f495c1270b189446a"
server_key_iv: "d34a11ea326025df1a0f197e650985ad"
server_key_version: "encryptionkey@0"
I20260925 19:05:05.500954 4334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:05.517757 4334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:05.520350 4334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:05.520551 4334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:05.524962 4334 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:45225
I20260925 19:05:05.525048 4401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:45225 every 8 connection(s)
I20260925 19:05:05.525458 4334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:05:05.526404 4402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:05.528812 4402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0251be0e0d48829fd052d81d67d6b0: Bootstrap starting.
I20260925 19:05:05.529397 4402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d0251be0e0d48829fd052d81d67d6b0: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:05.529649 4402 log.cc:824] T 00000000000000000000000000000000 P 3d0251be0e0d48829fd052d81d67d6b0: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:05.530372 4402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0251be0e0d48829fd052d81d67d6b0: No bootstrap required, opened a new log
I20260925 19:05:05.532047 4402 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d0251be0e0d48829fd052d81d67d6b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0251be0e0d48829fd052d81d67d6b0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 45225 } }
I20260925 19:05:05.532191 4402 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d0251be0e0d48829fd052d81d67d6b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:05.532215 4402 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d0251be0e0d48829fd052d81d67d6b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0251be0e0d48829fd052d81d67d6b0, State: Initialized, Role: FOLLOWER
I20260925 19:05:05.532357 4402 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d0251be0e0d48829fd052d81d67d6b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0251be0e0d48829fd052d81d67d6b0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 45225 } }
I20260925 19:05:05.532439 4402 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d0251be0e0d48829fd052d81d67d6b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:05:05.532461 4402 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d0251be0e0d48829fd052d81d67d6b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:05:05.532490 4402 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d0251be0e0d48829fd052d81d67d6b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:05.532992 4402 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d0251be0e0d48829fd052d81d67d6b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0251be0e0d48829fd052d81d67d6b0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 45225 } }
I20260925 19:05:05.533078 4402 leader_election.cc:302] T 00000000000000000000000000000000 P 3d0251be0e0d48829fd052d81d67d6b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d0251be0e0d48829fd052d81d67d6b0; no voters:
I20260925 19:05:05.533222 4402 leader_election.cc:288] T 00000000000000000000000000000000 P 3d0251be0e0d48829fd052d81d67d6b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:05:05.533310 4405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d0251be0e0d48829fd052d81d67d6b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:05:05.533516 4402 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d0251be0e0d48829fd052d81d67d6b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:05:05.533638 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 4334
I20260925 19:05:05.533761 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:05:05.533912 787 external_mini_cluster.cc:1428] Setting key aeef91b47f8471756376385a9ba36e40
I20260925 19:05:05.534417 4405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d0251be0e0d48829fd052d81d67d6b0 [term 1 LEADER]: Becoming Leader. State: Replica: 3d0251be0e0d48829fd052d81d67d6b0, State: Running, Role: LEADER
I20260925 19:05:05.534569 4405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d0251be0e0d48829fd052d81d67d6b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0251be0e0d48829fd052d81d67d6b0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 45225 } }
I20260925 19:05:05.535066 4412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0251be0e0d48829fd052d81d67d6b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d0251be0e0d48829fd052d81d67d6b0. Latest consensus state: current_term: 1 leader_uuid: "3d0251be0e0d48829fd052d81d67d6b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0251be0e0d48829fd052d81d67d6b0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 45225 } } }
I20260925 19:05:05.535158 4412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0251be0e0d48829fd052d81d67d6b0 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:05.535285 4406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0251be0e0d48829fd052d81d67d6b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d0251be0e0d48829fd052d81d67d6b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0251be0e0d48829fd052d81d67d6b0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 45225 } } }
I20260925 19:05:05.535369 4406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0251be0e0d48829fd052d81d67d6b0 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:05.535631 4421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:05:05.536381 4421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:05:05.538019 4421 catalog_manager.cc:1347] Generated new cluster ID: ae68bae3b6994ed1baf7219217dc0af1
I20260925 19:05:05.538084 4421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:05:05 dist-test-slave-rn98 krb5kdc[4321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:05:05.546496 4417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:05.534632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57598 (local address 127.0.196.254:45225)
0925 19:05:05.534827 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:05.534838 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:05.535232 (+ 394us) server_negotiation.cc:407] Connection header received
0925 19:05:05.535370 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:05.535378 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:05.535443 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:05.535590 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:05.536516 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:05.537878 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:05.538959 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:05.539613 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:05.539742 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:05.542642 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:05.542647 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:05.542648 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:05.542661 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:05.544767 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:05.545221 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:05.545225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:05.545225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:05.545342 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:05.545567 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:05.545568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:05.545568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:05.545874 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:05.546018 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:05.546136 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 19:05:05.546272 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":108,"threads_started":1}
I20260925 19:05:05.556485 4421 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:05:05.557299 4421 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:05:05.569025 4421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d0251be0e0d48829fd052d81d67d6b0: Generated new TSK 0
I20260925 19:05:05.569269 4421 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
I20260925 19:05:05.606454 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:0
--local_ip_for_outbound_sockets=127.0.196.193
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:45225
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:35945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:05.690970 4428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:05.691229 4428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:05.691272 4428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:05.692484 4428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:05.692561 4428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:05.692607 4428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:05:05.694168 4428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:35945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:45225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:05.694455 4428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:05.694734 4428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:05.700284 4433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:05.700354 4436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:05.700548 4434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:05.700728 4428 server_base.cc:1034] running on GCE node
I20260925 19:05:05.700974 4428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:05.701256 4428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:05.702442 4428 hybrid_clock.cc:648] HybridClock initialized: now 1790363105702417 us; error 50 us; skew 500 ppm
Sep 25 19:05:05 dist-test-slave-rn98 krb5kdc[4321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:05.705425 4428 init.cc:377] Logged in from keytab as kudu/127.0.196.193@KRBTEST.COM (short username kudu)
I20260925 19:05:05.705953 4428 webserver.cc:460] Webserver started at http://127.0.196.193:32957/ using document root <none> and password file <none>
I20260925 19:05:05.706194 4428 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:05.706243 4428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:05.706462 4428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:05.707647 4428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-0/data/instance:
uuid: "813873b1ca3449b8ae5ed24a56985de8"
format_stamp: "Formatted at 2026-09-25 19:05:05 on dist-test-slave-rn98"
server_key: "690e93aac5074de738c640efae5eb9b8"
server_key_iv: "83b6b044076bb336e16f4517f49fa8c7"
server_key_version: "encryptionkey@0"
I20260925 19:05:05.708016 4428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-0/wal/instance:
uuid: "813873b1ca3449b8ae5ed24a56985de8"
format_stamp: "Formatted at 2026-09-25 19:05:05 on dist-test-slave-rn98"
server_key: "690e93aac5074de738c640efae5eb9b8"
server_key_iv: "83b6b044076bb336e16f4517f49fa8c7"
server_key_version: "encryptionkey@0"
I20260925 19:05:05.709425 4428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:05:05.710198 4443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:05.710384 4428 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:05.710472 4428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "813873b1ca3449b8ae5ed24a56985de8"
format_stamp: "Formatted at 2026-09-25 19:05:05 on dist-test-slave-rn98"
server_key: "690e93aac5074de738c640efae5eb9b8"
server_key_iv: "83b6b044076bb336e16f4517f49fa8c7"
server_key_version: "encryptionkey@0"
I20260925 19:05:05.710526 4428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:05.727695 4428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:05.730161 4428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:05.730283 4428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:05.730499 4428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:05.730904 4428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:05.730943 4428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:05.730967 4428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:05.730980 4428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:05.736840 4428 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:32847
I20260925 19:05:05.736902 4556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:32847 every 8 connection(s)
I20260925 19:05:05.737247 4428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
I20260925 19:05:05.742054 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 4428
I20260925 19:05:05.742161 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-0/wal/instance
I20260925 19:05:05.742355 787 external_mini_cluster.cc:1428] Setting key 4324b980ef2d67cd12ec6ac584749392
Sep 25 19:05:05 dist-test-slave-rn98 krb5kdc[4321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:05:05.748210 4417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:05.738031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:40507 (local address 127.0.196.254:45225)
0925 19:05:05.738098 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:05.738100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:05.738283 (+ 183us) server_negotiation.cc:407] Connection header received
0925 19:05:05.739277 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:05.739278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:05.739292 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:05.739356 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:05.739994 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:05.740824 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:05.742038 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:05.742419 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:05.742505 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:05.745213 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:05.745215 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:05.745216 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:05.745224 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:05.746748 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:05.747197 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:05.747199 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:05.747199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:05.747265 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:05.747529 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:05.747530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:05.747531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:05.747658 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:05.747725 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:05.748024 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 19:05:05.748110 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 19:05:05.748448 4559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:05.738004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:45225 (local address 127.0.196.193:40507)
0925 19:05:05.738149 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:05.738192 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:05.739179 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:05.739413 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:05.739417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:05.739703 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:05.739906 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:05.739910 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:05.740967 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:05.740969 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:05.741949 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:05.741952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:05.742518 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:05.742520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:05.742707 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:05.743418 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:05.743427 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:05.745096 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:05.746845 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:05.746852 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:05.746854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:05.747095 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:05.747330 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:05.747331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:05.747331 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:05.747460 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:05.747729 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:05.747735 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:05.747889 (+ 154us) client_negotiation.cc:769] Sending connection context
0925 19:05:05.748003 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 19:05:05.748186 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260925 19:05:05.749017 4557 heartbeater.cc:347] Connected to a master server at 127.0.196.254:45225
I20260925 19:05:05.749136 4557 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:05.749392 4557 heartbeater.cc:511] Master 127.0.196.254:45225 requested a full tablet report, sending...
I20260925 19:05:05.749953 4366 ts_manager.cc:194] Registered new tserver with Master: 813873b1ca3449b8ae5ed24a56985de8 (127.0.196.193:32847)
WARNING: no policy specified for kudu/127.0.196.194@KRBTEST.COM; defaulting to no policy
I20260925 19:05:05.750804 4366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.193@KRBTEST.COM'} at 127.0.196.193:40507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
I20260925 19:05:05.770949 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:0
--local_ip_for_outbound_sockets=127.0.196.194
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:45225
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:35945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:05.855713 4562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:05.855928 4562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:05.855953 4562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:05.857125 4562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:05.857170 4562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:05.857197 4562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:05:05.858767 4562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:35945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:45225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:05.859001 4562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:05.859202 4562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:05.864462 4568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:05.864477 4570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:05.864606 4562 server_base.cc:1034] running on GCE node
W20260925 19:05:05.864717 4567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:05.864946 4562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:05.865173 4562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:05.866321 4562 hybrid_clock.cc:648] HybridClock initialized: now 1790363105866310 us; error 32 us; skew 500 ppm
Sep 25 19:05:05 dist-test-slave-rn98 krb5kdc[4321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:05.868996 4562 init.cc:377] Logged in from keytab as kudu/127.0.196.194@KRBTEST.COM (short username kudu)
I20260925 19:05:05.869490 4562 webserver.cc:460] Webserver started at http://127.0.196.194:36887/ using document root <none> and password file <none>
I20260925 19:05:05.869740 4562 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:05.869787 4562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:05.869952 4562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:05.871130 4562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-1/data/instance:
uuid: "eb6b0c7204b149369742f225f0b81295"
format_stamp: "Formatted at 2026-09-25 19:05:05 on dist-test-slave-rn98"
server_key: "a2ec9ebe8f8588742dabe030080b4ed1"
server_key_iv: "a575465e1fb2f366be3d0103e9ff420f"
server_key_version: "encryptionkey@0"
I20260925 19:05:05.871487 4562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-1/wal/instance:
uuid: "eb6b0c7204b149369742f225f0b81295"
format_stamp: "Formatted at 2026-09-25 19:05:05 on dist-test-slave-rn98"
server_key: "a2ec9ebe8f8588742dabe030080b4ed1"
server_key_iv: "a575465e1fb2f366be3d0103e9ff420f"
server_key_version: "encryptionkey@0"
I20260925 19:05:05.872848 4562 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 19:05:05.873719 4577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:05.873967 4562 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:05.874150 4562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "eb6b0c7204b149369742f225f0b81295"
format_stamp: "Formatted at 2026-09-25 19:05:05 on dist-test-slave-rn98"
server_key: "a2ec9ebe8f8588742dabe030080b4ed1"
server_key_iv: "a575465e1fb2f366be3d0103e9ff420f"
server_key_version: "encryptionkey@0"
I20260925 19:05:05.874219 4562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:05.883112 4562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:05.885497 4562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:05.885679 4562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:05.885921 4562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:05.886312 4562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:05.886348 4562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:05.886371 4562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:05.886384 4562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:05.892661 4562 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:45739
I20260925 19:05:05.892706 4690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:45739 every 8 connection(s)
I20260925 19:05:05.893047 4562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
I20260925 19:05:05.896360 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 4562
I20260925 19:05:05.896476 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-1/wal/instance
I20260925 19:05:05.896664 787 external_mini_cluster.cc:1428] Setting key 88c6b494a5afa25e0781ca1a222164fb
Sep 25 19:05:05 dist-test-slave-rn98 krb5kdc[4321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:05:05.904199 4417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:05.893818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:46009 (local address 127.0.196.254:45225)
0925 19:05:05.893896 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:05.893898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:05.894146 (+ 248us) server_negotiation.cc:407] Connection header received
0925 19:05:05.895259 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:05.895261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:05.895275 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:05.895337 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:05.896000 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:05.896931 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:05.898046 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:05.898274 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:05.898354 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:05.900798 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:05.900800 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:05.900800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:05.900810 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:05.902535 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:05.903055 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:05.903057 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:05.903058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:05.903131 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:05.903456 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:05.903457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:05.903458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:05.903611 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:05.903688 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:05.903976 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 19:05:05.904083 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 19:05:05.904310 4693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:05.893823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:45225 (local address 127.0.196.194:46009)
0925 19:05:05.893997 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:05.894035 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:05.895138 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:05.895414 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:05.895418 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:05.895695 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:05.895898 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:05.895901 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:05.897086 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:05.897087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:05.897942 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:05.897945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:05.898370 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:05.898371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:05.898507 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:05.899191 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:05.899200 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:05.900662 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:05.902732 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:05.902739 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:05.902741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:05.902975 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:05.903247 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:05.903248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:05.903248 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:05.903354 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:05.903688 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:05.903692 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:05.903821 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 19:05:05.903920 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 19:05:05.904074 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":89,"threads_started":1}
I20260925 19:05:05.904778 4691 heartbeater.cc:347] Connected to a master server at 127.0.196.254:45225
I20260925 19:05:05.904899 4691 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:05.905138 4691 heartbeater.cc:511] Master 127.0.196.254:45225 requested a full tablet report, sending...
I20260925 19:05:05.905524 4366 ts_manager.cc:194] Registered new tserver with Master: eb6b0c7204b149369742f225f0b81295 (127.0.196.194:45739)
WARNING: no policy specified for kudu/127.0.196.195@KRBTEST.COM; defaulting to no policy
I20260925 19:05:05.905906 4366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.194@KRBTEST.COM'} at 127.0.196.194:46009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
I20260925 19:05:05.927224 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:0
--local_ip_for_outbound_sockets=127.0.196.195
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:45225
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:35945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:06.015949 4696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:06.016330 4696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:06.016378 4696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:06.018589 4696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:06.018708 4696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:06.018791 4696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:05:06.021050 4696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:35945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:45225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:06.021351 4696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:06.021636 4696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:06.026989 4704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:06.026996 4702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:06.026996 4701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:06.027261 4696 server_base.cc:1034] running on GCE node
I20260925 19:05:06.027513 4696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:06.027794 4696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:06.028975 4696 hybrid_clock.cc:648] HybridClock initialized: now 1790363106028959 us; error 33 us; skew 500 ppm
Sep 25 19:05:06 dist-test-slave-rn98 krb5kdc[4321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:06.031698 4696 init.cc:377] Logged in from keytab as kudu/127.0.196.195@KRBTEST.COM (short username kudu)
I20260925 19:05:06.032220 4696 webserver.cc:460] Webserver started at http://127.0.196.195:40877/ using document root <none> and password file <none>
I20260925 19:05:06.032490 4696 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:06.032542 4696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:06.032687 4696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:06.033933 4696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-2/data/instance:
uuid: "feb6643787f54c8689de56427b5204f4"
format_stamp: "Formatted at 2026-09-25 19:05:06 on dist-test-slave-rn98"
server_key: "2df19aab30523c21e1ec052966d067b6"
server_key_iv: "b46e97bd10eaa40e78b7d6c125d2583a"
server_key_version: "encryptionkey@0"
I20260925 19:05:06.034368 4696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-2/wal/instance:
uuid: "feb6643787f54c8689de56427b5204f4"
format_stamp: "Formatted at 2026-09-25 19:05:06 on dist-test-slave-rn98"
server_key: "2df19aab30523c21e1ec052966d067b6"
server_key_iv: "b46e97bd10eaa40e78b7d6c125d2583a"
server_key_version: "encryptionkey@0"
I20260925 19:05:06.036109 4696 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 19:05:06.036971 4711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:06.037169 4696 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:06.037303 4696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "feb6643787f54c8689de56427b5204f4"
format_stamp: "Formatted at 2026-09-25 19:05:06 on dist-test-slave-rn98"
server_key: "2df19aab30523c21e1ec052966d067b6"
server_key_iv: "b46e97bd10eaa40e78b7d6c125d2583a"
server_key_version: "encryptionkey@0"
I20260925 19:05:06.037385 4696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:06.048883 4696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:06.050925 4696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:06.051076 4696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:06.051357 4696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:06.051681 4696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:06.051739 4696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:06.051795 4696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:06.051824 4696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:06.058020 4696 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:39749
I20260925 19:05:06.058110 4824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:39749 every 8 connection(s)
I20260925 19:05:06.058475 4696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
I20260925 19:05:06.062542 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 4696
I20260925 19:05:06.062701 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790363087086945-787-0/minicluster-data/ts-2/wal/instance
I20260925 19:05:06.062880 787 external_mini_cluster.cc:1428] Setting key 07dbb0811a78160bcbc62f034cfa4d9c
Sep 25 19:05:06 dist-test-slave-rn98 krb5kdc[4321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:05:06.069697 4417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:06.059439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:57891 (local address 127.0.196.254:45225)
0925 19:05:06.059545 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:06.059548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:06.059767 (+ 219us) server_negotiation.cc:407] Connection header received
0925 19:05:06.060954 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:06.060956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:06.060970 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:06.061045 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:06.061996 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:06.062524 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:06.063982 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:06.064194 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:06.064282 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:06.066602 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:06.066603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:06.066604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:06.066612 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:06.068133 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:06.068589 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:06.068591 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:06.068591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:06.068660 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:06.068960 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:06.068963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:06.068963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:06.069091 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:06.069173 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:06.069493 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 19:05:06.069590 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 19:05:06.069878 4827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:06.059361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:45225 (local address 127.0.196.195:57891)
0925 19:05:06.059610 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:06.059654 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:06.060813 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:06.061141 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:06.061145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:06.061638 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:06.061853 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:06.061859 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:06.062646 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:06.062647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:06.063884 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:06.063887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:06.064293 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:06.064294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:06.064436 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:06.065025 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:06.065042 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:06.066483 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:06.068253 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:06.068261 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:06.068264 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:06.068506 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:06.068735 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:06.068736 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:06.068736 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:06.068862 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:06.069165 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:06.069172 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:06.069330 (+ 158us) client_negotiation.cc:769] Sending connection context
0925 19:05:06.069451 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 19:05:06.069623 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":93,"threads_started":1}
I20260925 19:05:06.070410 4825 heartbeater.cc:347] Connected to a master server at 127.0.196.254:45225
I20260925 19:05:06.070509 4825 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:06.070784 4825 heartbeater.cc:511] Master 127.0.196.254:45225 requested a full tablet report, sending...
I20260925 19:05:06.071264 4366 ts_manager.cc:194] Registered new tserver with Master: feb6643787f54c8689de56427b5204f4 (127.0.196.195:39749)
I20260925 19:05:06.071691 4366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.195@KRBTEST.COM'} at 127.0.196.195:57891
I20260925 19:05:06.075103 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 19:05:06 dist-test-slave-rn98 krb5kdc[4321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 19:05:06 dist-test-slave-rn98 krb5kdc[4321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.193@KRBTEST.COM
I20260925 19:05:06.100445 4832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:06.091549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40038 (local address 127.0.196.193:32847)
0925 19:05:06.091768 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:06.091774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:06.091807 (+ 33us) server_negotiation.cc:407] Connection header received
0925 19:05:06.091835 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:06.091837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:06.091852 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:06.091917 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:06.092589 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:06.093316 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:06.094237 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:06.094833 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:06.094908 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:06.097765 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:06.097775 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:06.097775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:06.097782 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:06.099323 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:06.099670 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:06.099674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:06.099674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:06.099748 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:06.099926 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:06.099927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:06.099927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:06.100070 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:06.100179 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:06.100243 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 19:05:06.100312 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":114,"threads_started":1}
Sep 25 19:05:06 dist-test-slave-rn98 krb5kdc[4321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:05:06.109688 4417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:06.101960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57600 (local address 127.0.196.254:45225)
0925 19:05:06.102003 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:06.102005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:06.102159 (+ 154us) server_negotiation.cc:407] Connection header received
0925 19:05:06.102190 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:06.102191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:06.102201 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:06.102261 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:06.102832 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:06.103400 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:06.104179 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:06.104389 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:06.104463 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:06.106932 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:06.106934 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:06.106934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:06.106942 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:06.108466 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:06.108792 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:06.108793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:06.108793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:06.108847 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:06.109050 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:06.109052 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:06.109052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:06.109241 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:06.109338 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:06.109422 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 19:05:06.109491 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 19:05:06.109962 4366 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57600
I20260925 19:05:06.110320 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 4428
I20260925 19:05:06.115465 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 4562
I20260925 19:05:06.120507 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 4696
I20260925 19:05:06.124924 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 4334
2026-09-25T19:05:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1906 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:05:06 dist-test-slave-rn98 krb5kdc[4837](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:05:06 dist-test-slave-rn98 krb5kdc[4837](info): set up 2 sockets
Sep 25 19:05:06 dist-test-slave-rn98 krb5kdc[4837](info): commencing operation
krb5kdc: starting...
W20260925 19:05:07.194829 787 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:05:07 dist-test-slave-rn98 krb5kdc[4837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:05:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:05:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:05:07.322098 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:39035
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:36295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:39035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:07.404834 4850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:07.405050 4850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:07.405074 4850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:07.406273 4850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:05:07.406319 4850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:07.406333 4850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:07.406347 4850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:05:07.406359 4850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:05:07.407974 4850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:36295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:39035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:39035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:07.408243 4850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:07.408497 4850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:07.413765 4855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:07.413900 4858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:07.413941 4850 server_base.cc:1034] running on GCE node
W20260925 19:05:07.413774 4856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:07.414306 4850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:07.414700 4850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:07.415864 4850 hybrid_clock.cc:648] HybridClock initialized: now 1790363107415843 us; error 37 us; skew 500 ppm
Sep 25 19:05:07 dist-test-slave-rn98 krb5kdc[4837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:07.418497 4850 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:05:07.418886 4850 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:07.418936 4850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:07.419028 4850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:07.420061 4850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "725d6b293a2d4dbd9042a41e0a46546f"
format_stamp: "Formatted at 2026-09-25 19:05:07 on dist-test-slave-rn98"
server_key: "34631e023589bbe2f5cd871d50df35fa"
server_key_iv: "77ab871a09ac279c5f848c7c932f2fe3"
server_key_version: "encryptionkey@0"
I20260925 19:05:07.420377 4850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "725d6b293a2d4dbd9042a41e0a46546f"
format_stamp: "Formatted at 2026-09-25 19:05:07 on dist-test-slave-rn98"
server_key: "34631e023589bbe2f5cd871d50df35fa"
server_key_iv: "77ab871a09ac279c5f848c7c932f2fe3"
server_key_version: "encryptionkey@0"
I20260925 19:05:07.421649 4850 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:05:07.422385 4864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:07.422554 4850 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:05:07.422693 4850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "725d6b293a2d4dbd9042a41e0a46546f"
format_stamp: "Formatted at 2026-09-25 19:05:07 on dist-test-slave-rn98"
server_key: "34631e023589bbe2f5cd871d50df35fa"
server_key_iv: "77ab871a09ac279c5f848c7c932f2fe3"
server_key_version: "encryptionkey@0"
I20260925 19:05:07.422765 4850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:07.442430 4850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:07.445554 4850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:07.445742 4850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:07.451063 4850 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:39035
I20260925 19:05:07.451087 4916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:39035 every 8 connection(s)
I20260925 19:05:07.451447 4850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:05:07.452450 4917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:07.454911 4917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 725d6b293a2d4dbd9042a41e0a46546f: Bootstrap starting.
I20260925 19:05:07.455540 4917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 725d6b293a2d4dbd9042a41e0a46546f: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:07.455829 4917 log.cc:824] T 00000000000000000000000000000000 P 725d6b293a2d4dbd9042a41e0a46546f: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:07.456670 4917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 725d6b293a2d4dbd9042a41e0a46546f: No bootstrap required, opened a new log
I20260925 19:05:07.457190 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 4850
I20260925 19:05:07.457274 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:05:07.457476 787 external_mini_cluster.cc:1428] Setting key 1e4934281fa391c8dfe7ad377af51fd0
I20260925 19:05:07.459244 4917 raft_consensus.cc:348] T 00000000000000000000000000000000 P 725d6b293a2d4dbd9042a41e0a46546f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "725d6b293a2d4dbd9042a41e0a46546f" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 39035 } }
I20260925 19:05:07.459450 4917 raft_consensus.cc:374] T 00000000000000000000000000000000 P 725d6b293a2d4dbd9042a41e0a46546f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:07.459492 4917 raft_consensus.cc:729] T 00000000000000000000000000000000 P 725d6b293a2d4dbd9042a41e0a46546f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 725d6b293a2d4dbd9042a41e0a46546f, State: Initialized, Role: FOLLOWER
I20260925 19:05:07.459643 4917 consensus_queue.cc:260] T 00000000000000000000000000000000 P 725d6b293a2d4dbd9042a41e0a46546f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "725d6b293a2d4dbd9042a41e0a46546f" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 39035 } }
I20260925 19:05:07.459730 4917 raft_consensus.cc:388] T 00000000000000000000000000000000 P 725d6b293a2d4dbd9042a41e0a46546f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:05:07.459764 4917 raft_consensus.cc:482] T 00000000000000000000000000000000 P 725d6b293a2d4dbd9042a41e0a46546f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:05:07.459817 4917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 725d6b293a2d4dbd9042a41e0a46546f [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:07.460474 4917 raft_consensus.cc:504] T 00000000000000000000000000000000 P 725d6b293a2d4dbd9042a41e0a46546f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "725d6b293a2d4dbd9042a41e0a46546f" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 39035 } }
I20260925 19:05:07.460601 4917 leader_election.cc:302] T 00000000000000000000000000000000 P 725d6b293a2d4dbd9042a41e0a46546f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 725d6b293a2d4dbd9042a41e0a46546f; no voters:
I20260925 19:05:07.460814 4917 leader_election.cc:288] T 00000000000000000000000000000000 P 725d6b293a2d4dbd9042a41e0a46546f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:05:07.460839 4922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 725d6b293a2d4dbd9042a41e0a46546f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:05:07.460944 4922 raft_consensus.cc:686] T 00000000000000000000000000000000 P 725d6b293a2d4dbd9042a41e0a46546f [term 1 LEADER]: Becoming Leader. State: Replica: 725d6b293a2d4dbd9042a41e0a46546f, State: Running, Role: LEADER
I20260925 19:05:07.461025 4922 consensus_queue.cc:237] T 00000000000000000000000000000000 P 725d6b293a2d4dbd9042a41e0a46546f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "725d6b293a2d4dbd9042a41e0a46546f" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 39035 } }
I20260925 19:05:07.461081 4917 sys_catalog.cc:564] T 00000000000000000000000000000000 P 725d6b293a2d4dbd9042a41e0a46546f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:05:07.461355 4922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 725d6b293a2d4dbd9042a41e0a46546f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "725d6b293a2d4dbd9042a41e0a46546f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "725d6b293a2d4dbd9042a41e0a46546f" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 39035 } } }
I20260925 19:05:07.461426 4922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 725d6b293a2d4dbd9042a41e0a46546f [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:07.461550 4922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 725d6b293a2d4dbd9042a41e0a46546f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 725d6b293a2d4dbd9042a41e0a46546f. Latest consensus state: current_term: 1 leader_uuid: "725d6b293a2d4dbd9042a41e0a46546f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "725d6b293a2d4dbd9042a41e0a46546f" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 39035 } } }
I20260925 19:05:07.461614 4922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 725d6b293a2d4dbd9042a41e0a46546f [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:07.461798 4927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:05:07.462338 4927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 19:05:07 dist-test-slave-rn98 krb5kdc[4837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:05:07.466660 4927 catalog_manager.cc:1347] Generated new cluster ID: 4e2582d9cbb24f67b75c4c7bd3dfe663
I20260925 19:05:07.466718 4927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:05:07.469770 4921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:07.458061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57904 (local address 127.0.196.254:39035)
0925 19:05:07.458235 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:07.458243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:07.458268 (+ 25us) server_negotiation.cc:407] Connection header received
0925 19:05:07.459168 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:07.459176 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:07.459227 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:07.459362 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:07.460185 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:07.461382 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:07.462557 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:07.463084 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:07.463228 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:07.465890 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:07.465894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:07.465894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:07.465906 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:07.467959 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:07.468355 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:07.468358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:07.468358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:07.468459 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:07.468679 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:07.468679 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:07.468680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:07.468926 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:07.469455 (+ 529us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:07.469515 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 19:05:07.469624 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
I20260925 19:05:07.494210 4927 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:05:07.495018 4927 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:05:07.504618 4927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 725d6b293a2d4dbd9042a41e0a46546f: Generated new TSK 0
I20260925 19:05:07.504844 4927 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
I20260925 19:05:07.537479 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:0
--local_ip_for_outbound_sockets=127.0.196.193
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:39035
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:36295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:07.622175 4943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:07.622395 4943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:07.622422 4943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:07.623687 4943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:07.623739 4943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:07.623796 4943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:05:07.625406 4943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:36295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:39035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:07.625707 4943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:07.625959 4943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:07.631028 4949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:07.631016 4948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:07.631139 4951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:07.631250 4943 server_base.cc:1034] running on GCE node
I20260925 19:05:07.631539 4943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:07.631799 4943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:07.632972 4943 hybrid_clock.cc:648] HybridClock initialized: now 1790363107632880 us; error 133 us; skew 500 ppm
Sep 25 19:05:07 dist-test-slave-rn98 krb5kdc[4837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:07.635694 4943 init.cc:377] Logged in from keytab as kudu/127.0.196.193@KRBTEST.COM (short username kudu)
I20260925 19:05:07.636282 4943 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:07.636389 4943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:07.636561 4943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:07.637684 4943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-0/data/instance:
uuid: "f6fe81109eeb46ec94cb1cc2c7b8a0b1"
format_stamp: "Formatted at 2026-09-25 19:05:07 on dist-test-slave-rn98"
server_key: "cb6286591d12c4a8cf09b4dc2f82a577"
server_key_iv: "6b6c065163ba92e3aa1ab79cd50b380a"
server_key_version: "encryptionkey@0"
I20260925 19:05:07.638082 4943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-0/wal/instance:
uuid: "f6fe81109eeb46ec94cb1cc2c7b8a0b1"
format_stamp: "Formatted at 2026-09-25 19:05:07 on dist-test-slave-rn98"
server_key: "cb6286591d12c4a8cf09b4dc2f82a577"
server_key_iv: "6b6c065163ba92e3aa1ab79cd50b380a"
server_key_version: "encryptionkey@0"
I20260925 19:05:07.639585 4943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:05:07.640434 4957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:07.640627 4943 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:05:07.640739 4943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "f6fe81109eeb46ec94cb1cc2c7b8a0b1"
format_stamp: "Formatted at 2026-09-25 19:05:07 on dist-test-slave-rn98"
server_key: "cb6286591d12c4a8cf09b4dc2f82a577"
server_key_iv: "6b6c065163ba92e3aa1ab79cd50b380a"
server_key_version: "encryptionkey@0"
I20260925 19:05:07.640820 4943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:07.651383 4943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:07.653883 4943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:07.654055 4943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:07.654313 4943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:07.654717 4943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:07.654774 4943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:07.654815 4943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:07.654839 4943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:07.660790 4943 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:45253
I20260925 19:05:07.660835 5070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:45253 every 8 connection(s)
I20260925 19:05:07.661156 4943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
I20260925 19:05:07.662772 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 4943
I20260925 19:05:07.662905 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-0/wal/instance
I20260925 19:05:07.663100 787 external_mini_cluster.cc:1428] Setting key e148ac733738ee82e5239ef605a88f5d
Sep 25 19:05:07 dist-test-slave-rn98 krb5kdc[4837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.196.194@KRBTEST.COM; defaulting to no policy
I20260925 19:05:07.672037 4921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:07.661867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:37211 (local address 127.0.196.254:39035)
0925 19:05:07.661915 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:07.661917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:07.662222 (+ 305us) server_negotiation.cc:407] Connection header received
0925 19:05:07.662876 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:07.662878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:07.662894 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:07.662953 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:07.664397 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:07.664920 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:07.666078 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:07.666477 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:07.666558 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:07.668901 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:07.668903 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:07.668903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:07.668912 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:07.670563 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:07.671000 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:07.671001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:07.671002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:07.671066 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:07.671326 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:07.671327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:07.671327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:07.671467 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:07.671531 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:07.671794 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 19:05:07.671890 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:05:07.672075 5073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:07.661906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:39035 (local address 127.0.196.193:37211)
0925 19:05:07.662081 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:07.662125 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:07.662731 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:07.663156 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:07.663161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:07.663543 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:07.664243 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:07.664251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:07.665065 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:07.665066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:07.665979 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:07.665981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:07.666590 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:07.666591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:07.666794 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:07.667411 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:07.667419 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:07.668797 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:07.670694 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:07.670700 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:07.670702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:07.670920 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:07.671149 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:07.671150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:07.671150 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:07.671248 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:07.671528 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:07.671532 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:07.671647 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 19:05:07.671722 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 19:05:07.671900 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
I20260925 19:05:07.672595 5071 heartbeater.cc:347] Connected to a master server at 127.0.196.254:39035
I20260925 19:05:07.672706 5071 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:07.672875 5071 heartbeater.cc:511] Master 127.0.196.254:39035 requested a full tablet report, sending...
I20260925 19:05:07.673391 4877 ts_manager.cc:194] Registered new tserver with Master: f6fe81109eeb46ec94cb1cc2c7b8a0b1 (127.0.196.193:45253)
I20260925 19:05:07.674137 4877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.193@KRBTEST.COM'} at 127.0.196.193:37211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
I20260925 19:05:07.690462 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:0
--local_ip_for_outbound_sockets=127.0.196.194
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:39035
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:36295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:07.772078 5076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:07.772290 5076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:07.772315 5076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:07.773483 5076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:07.773526 5076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:07.773554 5076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:05:07.775086 5076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:36295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:39035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:07.775328 5076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:07.775580 5076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:07.780689 5082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:07.780683 5084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:07.780687 5081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:07.780941 5076 server_base.cc:1034] running on GCE node
I20260925 19:05:07.781211 5076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:07.781463 5076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:07.782606 5076 hybrid_clock.cc:648] HybridClock initialized: now 1790363107782596 us; error 29 us; skew 500 ppm
Sep 25 19:05:07 dist-test-slave-rn98 krb5kdc[4837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:07.785197 5076 init.cc:377] Logged in from keytab as kudu/127.0.196.194@KRBTEST.COM (short username kudu)
I20260925 19:05:07.785521 5076 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:07.785565 5076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:07.785657 5076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:07.786747 5076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-1/data/instance:
uuid: "a1e63ac2a73a4abfa32ba8ee84f29d96"
format_stamp: "Formatted at 2026-09-25 19:05:07 on dist-test-slave-rn98"
server_key: "d26ad45a5ef0949c8bf36ac2ec917b8b"
server_key_iv: "25d3b884dd91d5bc9fe28f566a2e6b5c"
server_key_version: "encryptionkey@0"
I20260925 19:05:07.787060 5076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-1/wal/instance:
uuid: "a1e63ac2a73a4abfa32ba8ee84f29d96"
format_stamp: "Formatted at 2026-09-25 19:05:07 on dist-test-slave-rn98"
server_key: "d26ad45a5ef0949c8bf36ac2ec917b8b"
server_key_iv: "25d3b884dd91d5bc9fe28f566a2e6b5c"
server_key_version: "encryptionkey@0"
I20260925 19:05:07.788331 5076 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:07.789076 5090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:07.789253 5076 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:07.789361 5076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "a1e63ac2a73a4abfa32ba8ee84f29d96"
format_stamp: "Formatted at 2026-09-25 19:05:07 on dist-test-slave-rn98"
server_key: "d26ad45a5ef0949c8bf36ac2ec917b8b"
server_key_iv: "25d3b884dd91d5bc9fe28f566a2e6b5c"
server_key_version: "encryptionkey@0"
I20260925 19:05:07.789438 5076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:07.806763 5076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:07.808997 5076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:07.809194 5076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:07.809446 5076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:07.809773 5076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:07.809829 5076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:07.809875 5076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:07.809898 5076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:07.815680 5076 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:39447
I20260925 19:05:07.815757 5203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:39447 every 8 connection(s)
I20260925 19:05:07.816051 5076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:05:07 dist-test-slave-rn98 krb5kdc[4837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:05:07.825551 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 5076
I20260925 19:05:07.825681 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-1/wal/instance
I20260925 19:05:07.825927 787 external_mini_cluster.cc:1428] Setting key f840fe7074dabeb6a1d940e8c6bb51a1
I20260925 19:05:07.826303 4921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:07.816811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:41877 (local address 127.0.196.254:39035)
0925 19:05:07.816872 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:07.816874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:07.817140 (+ 266us) server_negotiation.cc:407] Connection header received
0925 19:05:07.817660 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:07.817661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:07.817672 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:07.817726 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:07.818698 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:07.819208 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:07.820244 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:07.820441 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:07.820531 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:07.822991 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:07.822992 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:07.822993 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:07.822999 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:07.824647 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:07.825180 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:07.825182 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:07.825182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:07.825245 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:07.825501 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:07.825502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:07.825502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:07.825649 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:07.825713 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:07.826136 (+ 423us) server_negotiation.cc:299] Negotiation successful
0925 19:05:07.826212 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:05:07.826440 5206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:07.816782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:39035 (local address 127.0.196.194:41877)
0925 19:05:07.816962 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:07.816999 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:07.817523 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:07.817804 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:07.817808 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:07.818075 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:07.818549 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:07.818554 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:07.819318 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:07.819319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:07.820139 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:07.820141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:07.820542 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:07.820543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:07.820671 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:07.821227 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:07.821235 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:07.822867 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:07.824740 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:07.824746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:07.824748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:07.825078 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:07.825314 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:07.825316 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:07.825316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:07.825429 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:07.825750 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:07.825756 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:07.825887 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 19:05:07.826005 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 19:05:07.826191 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
I20260925 19:05:07.827068 5204 heartbeater.cc:347] Connected to a master server at 127.0.196.254:39035
I20260925 19:05:07.827181 5204 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:07.827381 5204 heartbeater.cc:511] Master 127.0.196.254:39035 requested a full tablet report, sending...
I20260925 19:05:07.827814 4877 ts_manager.cc:194] Registered new tserver with Master: a1e63ac2a73a4abfa32ba8ee84f29d96 (127.0.196.194:39447)
I20260925 19:05:07.828326 4877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.194@KRBTEST.COM'} at 127.0.196.194:41877
WARNING: no policy specified for kudu/127.0.196.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
I20260925 19:05:07.855299 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:0
--local_ip_for_outbound_sockets=127.0.196.195
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:39035
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:36295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:07.938758 5209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:07.938984 5209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:07.939018 5209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:07.940295 5209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:07.940346 5209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:07.940375 5209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:05:07.941970 5209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:36295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:39035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:07.942231 5209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:07.942500 5209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:07.947881 5214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:07.947953 5215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:07.947889 5217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:07.948323 5209 server_base.cc:1034] running on GCE node
I20260925 19:05:07.948558 5209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:07.948817 5209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:07.949985 5209 hybrid_clock.cc:648] HybridClock initialized: now 1790363107949979 us; error 36 us; skew 500 ppm
Sep 25 19:05:07 dist-test-slave-rn98 krb5kdc[4837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:07.952766 5209 init.cc:377] Logged in from keytab as kudu/127.0.196.195@KRBTEST.COM (short username kudu)
I20260925 19:05:07.953142 5209 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:07.953217 5209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:07.953377 5209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:07.954548 5209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-2/data/instance:
uuid: "4fb4fbed173b439e9bf0ae6525a6b344"
format_stamp: "Formatted at 2026-09-25 19:05:07 on dist-test-slave-rn98"
server_key: "f059fcd151360813641db70b4573ba6f"
server_key_iv: "7db51af16d92622858960e0ff0c4c31f"
server_key_version: "encryptionkey@0"
I20260925 19:05:07.955011 5209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-2/wal/instance:
uuid: "4fb4fbed173b439e9bf0ae6525a6b344"
format_stamp: "Formatted at 2026-09-25 19:05:07 on dist-test-slave-rn98"
server_key: "f059fcd151360813641db70b4573ba6f"
server_key_iv: "7db51af16d92622858960e0ff0c4c31f"
server_key_version: "encryptionkey@0"
I20260925 19:05:07.956411 5209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:07.957278 5223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:07.957479 5209 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:07.957599 5209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "4fb4fbed173b439e9bf0ae6525a6b344"
format_stamp: "Formatted at 2026-09-25 19:05:07 on dist-test-slave-rn98"
server_key: "f059fcd151360813641db70b4573ba6f"
server_key_iv: "7db51af16d92622858960e0ff0c4c31f"
server_key_version: "encryptionkey@0"
I20260925 19:05:07.957677 5209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:07.969043 5209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:07.971329 5209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:07.971496 5209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:07.971745 5209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:07.972061 5209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:07.972117 5209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:07.972160 5209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:07.972185 5209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:07.978147 5209 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:39337
I20260925 19:05:07.978178 5336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:39337 every 8 connection(s)
I20260925 19:05:07.978456 5209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
I20260925 19:05:07.980356 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 5209
I20260925 19:05:07.980425 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-2/wal/instance
I20260925 19:05:07.980633 787 external_mini_cluster.cc:1428] Setting key da73d6fb7b1c22394e379d216f599045
Sep 25 19:05:07 dist-test-slave-rn98 krb5kdc[4837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:05:07.988389 4921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:07.979215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:39725 (local address 127.0.196.254:39035)
0925 19:05:07.979276 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:07.979278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:07.979495 (+ 217us) server_negotiation.cc:407] Connection header received
0925 19:05:07.979984 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:07.979985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:07.979995 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:07.980059 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:07.981228 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:07.981871 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:07.983011 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:07.983216 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:07.983307 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:07.985528 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:07.985530 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:07.985531 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:07.985540 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:07.986901 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:07.987376 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:07.987377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:07.987378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:07.987441 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:07.987730 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:07.987731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:07.987731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:07.987850 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:07.987921 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:07.988215 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 19:05:07.988292 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:05:07.988621 5339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:07.979179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:39035 (local address 127.0.196.195:39725)
0925 19:05:07.979361 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:07.979404 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:07.979901 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:07.980130 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:07.980133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:07.980448 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:07.981148 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:07.981156 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:07.981965 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:07.981966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:07.982913 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:07.982916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:07.983318 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:07.983319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:07.983456 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:07.983958 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:07.983966 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:07.985427 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:07.987022 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:07.987029 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:07.987031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:07.987279 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:07.987526 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:07.987527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:07.987528 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:07.987651 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:07.987924 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:07.987930 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:07.988076 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 19:05:07.988184 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 19:05:07.988349 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":109,"threads_started":1}
I20260925 19:05:07.989297 5337 heartbeater.cc:347] Connected to a master server at 127.0.196.254:39035
I20260925 19:05:07.989383 5337 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:07.989509 5337 heartbeater.cc:511] Master 127.0.196.254:39035 requested a full tablet report, sending...
I20260925 19:05:07.989858 4877 ts_manager.cc:194] Registered new tserver with Master: 4fb4fbed173b439e9bf0ae6525a6b344 (127.0.196.195:39337)
I20260925 19:05:07.990283 4877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.195@KRBTEST.COM'} at 127.0.196.195:39725
I20260925 19:05:07.992751 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:05:08.000451 4921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:07.994115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57906 (local address 127.0.196.254:39035)
0925 19:05:07.994169 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:07.994171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:07.994281 (+ 110us) server_negotiation.cc:407] Connection header received
0925 19:05:07.994320 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:07.994321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:07.994329 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:07.994376 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:07.994950 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:07.995415 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:07.996165 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:07.996339 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:07.996392 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:07.997620 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:07.997621 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:07.997621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:07.997627 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:07.999147 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:07.999607 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:07.999609 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:07.999609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:07.999667 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:07.999869 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:07.999869 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:07.999870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:08.000017 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:08.000171 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:08.000315 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 19:05:08.000384 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 19:05:08.002283 4877 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:05:08.004492 4877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:05:08.011701 5351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:08.007327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57952 (local address 127.0.196.195:39337)
0925 19:05:08.007552 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:08.007556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:08.007678 (+ 122us) server_negotiation.cc:407] Connection header received
0925 19:05:08.007718 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:08.007720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:08.007735 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:08.007808 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:08.008022 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:08.008724 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:08.010717 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:08.011236 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:08.011333 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:08.011404 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:08.011574 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 19:05:08.011617 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":76,"threads_started":1}
I20260925 19:05:08.011689 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:08.007059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:39337 (local address 127.0.0.1:57952)
0925 19:05:08.007615 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:08.007622 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:08.007657 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:08.007835 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:08.007838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:08.007841 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:08.007937 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:08.007939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:08.009809 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:08.009810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:08.010585 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:08.010587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:08.011337 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:08.011338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:08.011425 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:08.011439 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 19:05:08.011552 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 19:05:08.011608 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":28,"thread_start_us":49,"threads_started":1}
I20260925 19:05:08.011920 5348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:08.007058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:39447 (local address 127.0.0.1:45938)
0925 19:05:08.007324 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:08.007366 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:08.007438 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:08.007836 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:08.007839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:08.007841 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:08.007933 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:08.007936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:08.008823 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:08.008824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:08.009652 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:08.009654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:08.011661 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:08.011662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:08.011726 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:08.011730 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:05:08.011777 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 19:05:08.011831 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":9,"thread_start_us":105,"threads_started":1}
I20260925 19:05:08.012233 5352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:08.007340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45938 (local address 127.0.196.194:39447)
0925 19:05:08.007544 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:08.007550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:08.007562 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:05:08.007601 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:08.007603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:08.007619 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:08.007733 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:08.008019 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:08.008727 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:08.010778 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:08.011504 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:08.011572 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:08.011626 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:08.012095 (+ 469us) server_negotiation.cc:299] Negotiation successful
0925 19:05:08.012139 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":95,"threads_started":1}
I20260925 19:05:08.012315 5350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:08.007058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:45253 (local address 127.0.0.1:48582)
0925 19:05:08.007487 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:08.007496 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:08.007535 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:08.009033 (+ 1498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:08.009034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:08.009036 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:08.009085 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:08.009087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:08.010042 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:08.010043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:08.011041 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:08.011044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:08.012108 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:08.012109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:08.012166 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:08.012169 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:05:08.012198 (+ 29us) client_negotiation.cc:240] Negotiation successful
0925 19:05:08.012236 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":2,"thread_start_us":48,"threads_started":1}
I20260925 19:05:08.012625 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:08.007340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48582 (local address 127.0.196.193:45253)
0925 19:05:08.008866 (+ 1526us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:08.008870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:08.008878 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:05:08.008917 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:08.008919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:08.008932 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:08.009031 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:08.009140 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:08.009946 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:08.011137 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:08.012017 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:08.012406 (+ 389us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:08.012469 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:08.012494 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 19:05:08.012546 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1495,"thread_start_us":90,"threads_started":1}
I20260925 19:05:08.012845 5138 tablet_service.cc:1467] Processing CreateTablet for tablet 7d94a88e27a6474d802d22aa98ea243a (DEFAULT_TABLE table=test-table [id=432b405f74de4336b98a6ee96ef547d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:05:08.012856 5271 tablet_service.cc:1467] Processing CreateTablet for tablet 7d94a88e27a6474d802d22aa98ea243a (DEFAULT_TABLE table=test-table [id=432b405f74de4336b98a6ee96ef547d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:05:08.013270 5138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d94a88e27a6474d802d22aa98ea243a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:08.013255 5271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d94a88e27a6474d802d22aa98ea243a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:08.015329 5005 tablet_service.cc:1467] Processing CreateTablet for tablet 7d94a88e27a6474d802d22aa98ea243a (DEFAULT_TABLE table=test-table [id=432b405f74de4336b98a6ee96ef547d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:05:08.015694 5005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d94a88e27a6474d802d22aa98ea243a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:08.016472 5354 tablet_bootstrap.cc:492] T 7d94a88e27a6474d802d22aa98ea243a P 4fb4fbed173b439e9bf0ae6525a6b344: Bootstrap starting.
I20260925 19:05:08.016873 5355 tablet_bootstrap.cc:492] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96: Bootstrap starting.
I20260925 19:05:08.017244 5354 tablet_bootstrap.cc:654] T 7d94a88e27a6474d802d22aa98ea243a P 4fb4fbed173b439e9bf0ae6525a6b344: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:08.017514 5355 tablet_bootstrap.cc:654] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:08.017942 5354 log.cc:824] T 7d94a88e27a6474d802d22aa98ea243a P 4fb4fbed173b439e9bf0ae6525a6b344: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:08.018151 5355 log.cc:824] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:08.018903 5354 tablet_bootstrap.cc:492] T 7d94a88e27a6474d802d22aa98ea243a P 4fb4fbed173b439e9bf0ae6525a6b344: No bootstrap required, opened a new log
I20260925 19:05:08.018947 5358 tablet_bootstrap.cc:492] T 7d94a88e27a6474d802d22aa98ea243a P f6fe81109eeb46ec94cb1cc2c7b8a0b1: Bootstrap starting.
I20260925 19:05:08.018973 5354 ts_tablet_manager.cc:1397] T 7d94a88e27a6474d802d22aa98ea243a P 4fb4fbed173b439e9bf0ae6525a6b344: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 19:05:08.019562 5358 tablet_bootstrap.cc:654] T 7d94a88e27a6474d802d22aa98ea243a P f6fe81109eeb46ec94cb1cc2c7b8a0b1: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:08.020219 5358 log.cc:824] T 7d94a88e27a6474d802d22aa98ea243a P f6fe81109eeb46ec94cb1cc2c7b8a0b1: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:08.021034 5358 tablet_bootstrap.cc:492] T 7d94a88e27a6474d802d22aa98ea243a P f6fe81109eeb46ec94cb1cc2c7b8a0b1: No bootstrap required, opened a new log
I20260925 19:05:08.021121 5358 ts_tablet_manager.cc:1397] T 7d94a88e27a6474d802d22aa98ea243a P f6fe81109eeb46ec94cb1cc2c7b8a0b1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:05:08.021164 5354 raft_consensus.cc:348] T 7d94a88e27a6474d802d22aa98ea243a P 4fb4fbed173b439e9bf0ae6525a6b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb4fbed173b439e9bf0ae6525a6b344" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39337 } } peers { permanent_uuid: "f6fe81109eeb46ec94cb1cc2c7b8a0b1" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45253 } } peers { permanent_uuid: "a1e63ac2a73a4abfa32ba8ee84f29d96" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 39447 } }
I20260925 19:05:08.021343 5354 raft_consensus.cc:374] T 7d94a88e27a6474d802d22aa98ea243a P 4fb4fbed173b439e9bf0ae6525a6b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:08.021382 5354 raft_consensus.cc:729] T 7d94a88e27a6474d802d22aa98ea243a P 4fb4fbed173b439e9bf0ae6525a6b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fb4fbed173b439e9bf0ae6525a6b344, State: Initialized, Role: FOLLOWER
I20260925 19:05:08.021503 5354 consensus_queue.cc:260] T 7d94a88e27a6474d802d22aa98ea243a P 4fb4fbed173b439e9bf0ae6525a6b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb4fbed173b439e9bf0ae6525a6b344" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39337 } } peers { permanent_uuid: "f6fe81109eeb46ec94cb1cc2c7b8a0b1" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45253 } } peers { permanent_uuid: "a1e63ac2a73a4abfa32ba8ee84f29d96" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 39447 } }
I20260925 19:05:08.021800 5354 ts_tablet_manager.cc:1428] T 7d94a88e27a6474d802d22aa98ea243a P 4fb4fbed173b439e9bf0ae6525a6b344: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 19:05:08.021819 5337 heartbeater.cc:503] Master 127.0.196.254:39035 was elected leader, sending a full tablet report...
I20260925 19:05:08.022501 5355 tablet_bootstrap.cc:492] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96: No bootstrap required, opened a new log
I20260925 19:05:08.022583 5355 ts_tablet_manager.cc:1397] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260925 19:05:08.023193 5358 raft_consensus.cc:348] T 7d94a88e27a6474d802d22aa98ea243a P f6fe81109eeb46ec94cb1cc2c7b8a0b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb4fbed173b439e9bf0ae6525a6b344" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39337 } } peers { permanent_uuid: "f6fe81109eeb46ec94cb1cc2c7b8a0b1" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45253 } } peers { permanent_uuid: "a1e63ac2a73a4abfa32ba8ee84f29d96" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 39447 } }
I20260925 19:05:08.023346 5358 raft_consensus.cc:374] T 7d94a88e27a6474d802d22aa98ea243a P f6fe81109eeb46ec94cb1cc2c7b8a0b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:08.023391 5358 raft_consensus.cc:729] T 7d94a88e27a6474d802d22aa98ea243a P f6fe81109eeb46ec94cb1cc2c7b8a0b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6fe81109eeb46ec94cb1cc2c7b8a0b1, State: Initialized, Role: FOLLOWER
I20260925 19:05:08.023512 5358 consensus_queue.cc:260] T 7d94a88e27a6474d802d22aa98ea243a P f6fe81109eeb46ec94cb1cc2c7b8a0b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb4fbed173b439e9bf0ae6525a6b344" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39337 } } peers { permanent_uuid: "f6fe81109eeb46ec94cb1cc2c7b8a0b1" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45253 } } peers { permanent_uuid: "a1e63ac2a73a4abfa32ba8ee84f29d96" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 39447 } }
I20260925 19:05:08.023800 5358 ts_tablet_manager.cc:1428] T 7d94a88e27a6474d802d22aa98ea243a P f6fe81109eeb46ec94cb1cc2c7b8a0b1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:05:08.023804 5071 heartbeater.cc:503] Master 127.0.196.254:39035 was elected leader, sending a full tablet report...
I20260925 19:05:08.024691 5355 raft_consensus.cc:348] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb4fbed173b439e9bf0ae6525a6b344" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39337 } } peers { permanent_uuid: "f6fe81109eeb46ec94cb1cc2c7b8a0b1" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45253 } } peers { permanent_uuid: "a1e63ac2a73a4abfa32ba8ee84f29d96" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 39447 } }
I20260925 19:05:08.024891 5355 raft_consensus.cc:374] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:08.024930 5355 raft_consensus.cc:729] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1e63ac2a73a4abfa32ba8ee84f29d96, State: Initialized, Role: FOLLOWER
I20260925 19:05:08.025046 5355 consensus_queue.cc:260] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb4fbed173b439e9bf0ae6525a6b344" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39337 } } peers { permanent_uuid: "f6fe81109eeb46ec94cb1cc2c7b8a0b1" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45253 } } peers { permanent_uuid: "a1e63ac2a73a4abfa32ba8ee84f29d96" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 39447 } }
I20260925 19:05:08.025333 5204 heartbeater.cc:503] Master 127.0.196.254:39035 was elected leader, sending a full tablet report...
I20260925 19:05:08.025312 5355 ts_tablet_manager.cc:1428] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260925 19:05:08.066586 5205 tablet.cc:2369] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:05:08.161872 5072 tablet.cc:2369] T 7d94a88e27a6474d802d22aa98ea243a P f6fe81109eeb46ec94cb1cc2c7b8a0b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:05:08.176821 5362 raft_consensus.cc:482] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:05:08.176995 5362 raft_consensus.cc:504] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb4fbed173b439e9bf0ae6525a6b344" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39337 } } peers { permanent_uuid: "f6fe81109eeb46ec94cb1cc2c7b8a0b1" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45253 } } peers { permanent_uuid: "a1e63ac2a73a4abfa32ba8ee84f29d96" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 39447 } }
I20260925 19:05:08.177459 5362 leader_election.cc:288] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fb4fbed173b439e9bf0ae6525a6b344 (127.0.196.195:39337), f6fe81109eeb46ec94cb1cc2c7b8a0b1 (127.0.196.193:45253)
I20260925 19:05:08.180701 5206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:08.177670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:39337 (local address 127.0.196.194:54741)
0925 19:05:08.177721 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:08.177734 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:08.177780 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:08.177980 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:08.177981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:08.177984 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:08.178062 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:08.178064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:08.178933 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:08.178934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:08.179767 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:08.179770 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:08.180372 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:08.180373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:08.180461 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:08.180465 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:05:08.180520 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 19:05:08.180556 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 19:05:08.180734 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:08.177754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:33675 (local address 127.0.196.193:45253)
0925 19:05:08.177801 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:08.177803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:08.178175 (+ 372us) server_negotiation.cc:407] Connection header received
0925 19:05:08.178207 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:08.178209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:08.178219 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:08.178274 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:08.178395 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:08.178952 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:08.179956 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:08.180314 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:08.180411 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:08.180425 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:08.180615 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 19:05:08.180658 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:05:08.180699 5351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:08.177775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:54741 (local address 127.0.196.195:39337)
0925 19:05:08.177816 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:08.177818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:08.177828 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:05:08.177865 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:08.177866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:08.177875 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:08.177950 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:08.178219 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:08.178832 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:08.179873 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:08.180287 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:08.180343 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:08.180350 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:08.180556 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 19:05:08.180615 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:05:08.181005 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:08.177672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:45253 (local address 127.0.196.194:33675)
0925 19:05:08.178070 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:08.178082 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:08.178146 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:08.178270 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:08.178270 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:08.178272 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:08.178333 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:08.178334 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:08.179031 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:08.179032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:08.179849 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:08.179851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:08.180430 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:08.180431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:08.180494 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:08.180497 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:05:08.180595 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 19:05:08.180634 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":161,"thread_start_us":148,"threads_started":1}
I20260925 19:05:08.181113 5291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d94a88e27a6474d802d22aa98ea243a" candidate_uuid: "a1e63ac2a73a4abfa32ba8ee84f29d96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fb4fbed173b439e9bf0ae6525a6b344" is_pre_election: true
I20260925 19:05:08.181303 5291 raft_consensus.cc:2454] T 7d94a88e27a6474d802d22aa98ea243a P 4fb4fbed173b439e9bf0ae6525a6b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1e63ac2a73a4abfa32ba8ee84f29d96 in term 0.
I20260925 19:05:08.181428 5025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d94a88e27a6474d802d22aa98ea243a" candidate_uuid: "a1e63ac2a73a4abfa32ba8ee84f29d96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6fe81109eeb46ec94cb1cc2c7b8a0b1" is_pre_election: true
I20260925 19:05:08.181520 5091 leader_election.cc:302] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fb4fbed173b439e9bf0ae6525a6b344, a1e63ac2a73a4abfa32ba8ee84f29d96; no voters:
I20260925 19:05:08.181598 5025 raft_consensus.cc:2454] T 7d94a88e27a6474d802d22aa98ea243a P f6fe81109eeb46ec94cb1cc2c7b8a0b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1e63ac2a73a4abfa32ba8ee84f29d96 in term 0.
I20260925 19:05:08.181665 5362 raft_consensus.cc:2781] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:05:08.181737 5362 raft_consensus.cc:482] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:05:08.181771 5362 raft_consensus.cc:3037] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:08.182435 5362 raft_consensus.cc:504] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb4fbed173b439e9bf0ae6525a6b344" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39337 } } peers { permanent_uuid: "f6fe81109eeb46ec94cb1cc2c7b8a0b1" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45253 } } peers { permanent_uuid: "a1e63ac2a73a4abfa32ba8ee84f29d96" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 39447 } }
I20260925 19:05:08.182595 5362 leader_election.cc:288] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96 [CANDIDATE]: Term 1 election: Requested vote from peers 4fb4fbed173b439e9bf0ae6525a6b344 (127.0.196.195:39337), f6fe81109eeb46ec94cb1cc2c7b8a0b1 (127.0.196.193:45253)
I20260925 19:05:08.182842 5025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d94a88e27a6474d802d22aa98ea243a" candidate_uuid: "a1e63ac2a73a4abfa32ba8ee84f29d96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6fe81109eeb46ec94cb1cc2c7b8a0b1"
I20260925 19:05:08.182868 5291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d94a88e27a6474d802d22aa98ea243a" candidate_uuid: "a1e63ac2a73a4abfa32ba8ee84f29d96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fb4fbed173b439e9bf0ae6525a6b344"
I20260925 19:05:08.182920 5025 raft_consensus.cc:3037] T 7d94a88e27a6474d802d22aa98ea243a P f6fe81109eeb46ec94cb1cc2c7b8a0b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:08.182962 5291 raft_consensus.cc:3037] T 7d94a88e27a6474d802d22aa98ea243a P 4fb4fbed173b439e9bf0ae6525a6b344 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:08.183676 5025 raft_consensus.cc:2454] T 7d94a88e27a6474d802d22aa98ea243a P f6fe81109eeb46ec94cb1cc2c7b8a0b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1e63ac2a73a4abfa32ba8ee84f29d96 in term 1.
I20260925 19:05:08.183830 5291 raft_consensus.cc:2454] T 7d94a88e27a6474d802d22aa98ea243a P 4fb4fbed173b439e9bf0ae6525a6b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1e63ac2a73a4abfa32ba8ee84f29d96 in term 1.
I20260925 19:05:08.183866 5093 leader_election.cc:302] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1e63ac2a73a4abfa32ba8ee84f29d96, f6fe81109eeb46ec94cb1cc2c7b8a0b1; no voters:
I20260925 19:05:08.183944 5362 raft_consensus.cc:2781] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:05:08.184118 5362 raft_consensus.cc:686] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96 [term 1 LEADER]: Becoming Leader. State: Replica: a1e63ac2a73a4abfa32ba8ee84f29d96, State: Running, Role: LEADER
I20260925 19:05:08.184227 5362 consensus_queue.cc:237] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb4fbed173b439e9bf0ae6525a6b344" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39337 } } peers { permanent_uuid: "f6fe81109eeb46ec94cb1cc2c7b8a0b1" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45253 } } peers { permanent_uuid: "a1e63ac2a73a4abfa32ba8ee84f29d96" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 39447 } }
I20260925 19:05:08.185072 4881 catalog_manager.cc:5495] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1e63ac2a73a4abfa32ba8ee84f29d96 (127.0.196.194). New cstate: current_term: 1 leader_uuid: "a1e63ac2a73a4abfa32ba8ee84f29d96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb4fbed173b439e9bf0ae6525a6b344" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6fe81109eeb46ec94cb1cc2c7b8a0b1" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1e63ac2a73a4abfa32ba8ee84f29d96" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 39447 } health_report { overall_health: HEALTHY } } }
I20260925 19:05:08.193740 5352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:08.191158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45942 (local address 127.0.196.194:39447)
0925 19:05:08.191224 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:08.191225 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:08.191232 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:05:08.191264 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:08.191265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:08.191281 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:08.191361 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:05:08.191471 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:08.191956 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:08.192847 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:08.193041 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:08.193183 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:08.193367 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:05:08.193482 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:05:08.193580 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:08.193599 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 19:05:08.193620 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 19:05:08.195868 5291 raft_consensus.cc:1261] T 7d94a88e27a6474d802d22aa98ea243a P 4fb4fbed173b439e9bf0ae6525a6b344 [term 1 FOLLOWER]: Refusing update from remote peer a1e63ac2a73a4abfa32ba8ee84f29d96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:05:08.196226 5025 raft_consensus.cc:1261] T 7d94a88e27a6474d802d22aa98ea243a P f6fe81109eeb46ec94cb1cc2c7b8a0b1 [term 1 FOLLOWER]: Refusing update from remote peer a1e63ac2a73a4abfa32ba8ee84f29d96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:05:08.196276 5364 consensus_queue.cc:1035] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fb4fbed173b439e9bf0ae6525a6b344" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 39337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:05:08.196549 5364 consensus_queue.cc:1035] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6fe81109eeb46ec94cb1cc2c7b8a0b1" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:05:08.198891 5368 mvcc.cc:205] Tried to move back new op lower bound from 7333327291166707712 to 7333327291123183616. Current Snapshot: MvccSnapshot[applied={T|T < 7333327291166707712}]
I20260925 19:05:08.199262 5369 mvcc.cc:205] Tried to move back new op lower bound from 7333327291166707712 to 7333327291123183616. Current Snapshot: MvccSnapshot[applied={T|T < 7333327291166707712}]
I20260925 19:05:08.201375 4877 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:05:08.201474 4877 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:05:08.202243 4877 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 725d6b293a2d4dbd9042a41e0a46546f: Sending DeleteTablet for 3 replicas of tablet 7d94a88e27a6474d802d22aa98ea243a
I20260925 19:05:08.202551 5271 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d94a88e27a6474d802d22aa98ea243a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:05:08 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:57952
I20260925 19:05:08.202569 5005 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d94a88e27a6474d802d22aa98ea243a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:05:08 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:48582
I20260925 19:05:08.202785 5138 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d94a88e27a6474d802d22aa98ea243a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:05:08 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:45938
I20260925 19:05:08.202827 5377 tablet_replica.cc:330] T 7d94a88e27a6474d802d22aa98ea243a P 4fb4fbed173b439e9bf0ae6525a6b344: stopping tablet replica
I20260925 19:05:08.202930 5377 raft_consensus.cc:2229] T 7d94a88e27a6474d802d22aa98ea243a P 4fb4fbed173b439e9bf0ae6525a6b344 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:05:08.203042 5377 raft_consensus.cc:2258] T 7d94a88e27a6474d802d22aa98ea243a P 4fb4fbed173b439e9bf0ae6525a6b344 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:05:08.203143 5379 tablet_replica.cc:330] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96: stopping tablet replica
I20260925 19:05:08.203227 5379 raft_consensus.cc:2229] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:05:08.203333 5379 raft_consensus.cc:2258] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:05:08.203338 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 4943
I20260925 19:05:08.207049 5377 ts_tablet_manager.cc:1905] T 7d94a88e27a6474d802d22aa98ea243a P 4fb4fbed173b439e9bf0ae6525a6b344: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:05:08.208531 5379 ts_tablet_manager.cc:1905] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 19:05:08.209307 4867 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.196.193:45253 (error 108)
I20260925 19:05:08.209848 5348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:08.209697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:45253 (local address 127.0.0.1:48596)
0925 19:05:08.209765 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:08.209811 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.193:45253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 19:05:08.210052 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 5076
I20260925 19:05:08.210108 5379 ts_tablet_manager.cc:1918] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260925 19:05:08.210103 4867 catalog_manager.cc:4584] TS f6fe81109eeb46ec94cb1cc2c7b8a0b1 (127.0.196.193:45253): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7d94a88e27a6474d802d22aa98ea243a: Network error: Client connection negotiation failed: client connection to 127.0.196.193:45253: connect: Connection refused (error 111)
I20260925 19:05:08.210176 5379 log.cc:1196] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-1/wal/wals/7d94a88e27a6474d802d22aa98ea243a
I20260925 19:05:08.210446 5379 ts_tablet_manager.cc:1939] T 7d94a88e27a6474d802d22aa98ea243a P a1e63ac2a73a4abfa32ba8ee84f29d96: Deleting consensus metadata
I20260925 19:05:08.210646 5377 ts_tablet_manager.cc:1918] T 7d94a88e27a6474d802d22aa98ea243a P 4fb4fbed173b439e9bf0ae6525a6b344: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:05:08.210726 5377 log.cc:1196] T 7d94a88e27a6474d802d22aa98ea243a P 4fb4fbed173b439e9bf0ae6525a6b344: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790363087086945-787-0/minicluster-data/ts-2/wal/wals/7d94a88e27a6474d802d22aa98ea243a
I20260925 19:05:08.210973 5377 ts_tablet_manager.cc:1939] T 7d94a88e27a6474d802d22aa98ea243a P 4fb4fbed173b439e9bf0ae6525a6b344: Deleting consensus metadata
I20260925 19:05:08.211432 4865 catalog_manager.cc:4841] TS 4fb4fbed173b439e9bf0ae6525a6b344 (127.0.196.195:39337): tablet 7d94a88e27a6474d802d22aa98ea243a (table test-table [id=432b405f74de4336b98a6ee96ef547d4]) successfully deleted
I20260925 19:05:08.216672 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 5209
I20260925 19:05:08.216779 5348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:08.216650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:39447 (local address 127.0.0.1:45952)
0925 19:05:08.216716 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:08.216749 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.194:39447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 19:05:08.221277 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 4850
2026-09-25T19:05:08Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2096 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T19:05:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:05:08Z Disabled control of system clock
I20260925 19:05:08.249440 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:42701
--webserver_interface=127.0.196.254
--webserver_port=0
--builtin_ntp_servers=127.0.196.212:33091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:42701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 19:05:08.332654 5384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:08.332882 5384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:08.332904 5384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:08.334169 5384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:05:08.334216 5384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:08.334229 5384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:08.334242 5384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:05:08.334254 5384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:05:08.335659 5384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:33091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:42701
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:42701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:08.335871 5384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:08.336180 5384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:08.341027 5392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:08.341058 5390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:08.341024 5389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:08.341331 5384 server_base.cc:1034] running on GCE node
I20260925 19:05:08.341640 5384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:08.341946 5384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:08.343109 5384 hybrid_clock.cc:648] HybridClock initialized: now 1790363108343102 us; error 36 us; skew 500 ppm
I20260925 19:05:08.344434 5384 webserver.cc:460] Webserver started at http://127.0.196.254:35387/ using document root <none> and password file <none>
I20260925 19:05:08.344698 5384 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:08.344772 5384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:08.344923 5384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:08.346043 5384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "38a4776bf9a24bdd955b1db12eb0034b"
format_stamp: "Formatted at 2026-09-25 19:05:08 on dist-test-slave-rn98"
server_key: "cb4e2939192e255186bbee7397419e6f"
server_key_iv: "29612af1de2cf4e35bfb31f7766aea36"
server_key_version: "encryptionkey@0"
I20260925 19:05:08.346426 5384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "38a4776bf9a24bdd955b1db12eb0034b"
format_stamp: "Formatted at 2026-09-25 19:05:08 on dist-test-slave-rn98"
server_key: "cb4e2939192e255186bbee7397419e6f"
server_key_iv: "29612af1de2cf4e35bfb31f7766aea36"
server_key_version: "encryptionkey@0"
I20260925 19:05:08.347914 5384 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:05:08.348776 5398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:08.349045 5384 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:05:08.349153 5384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "38a4776bf9a24bdd955b1db12eb0034b"
format_stamp: "Formatted at 2026-09-25 19:05:08 on dist-test-slave-rn98"
server_key: "cb4e2939192e255186bbee7397419e6f"
server_key_iv: "29612af1de2cf4e35bfb31f7766aea36"
server_key_version: "encryptionkey@0"
I20260925 19:05:08.349252 5384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:08.353067 5384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:08.353345 5384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:08.353507 5384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:08.357602 5384 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:42701
I20260925 19:05:08.357659 5450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:42701 every 8 connection(s)
I20260925 19:05:08.358053 5384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:05:08.358759 5451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:08.361667 5451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38a4776bf9a24bdd955b1db12eb0034b: Bootstrap starting.
I20260925 19:05:08.362269 5451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38a4776bf9a24bdd955b1db12eb0034b: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:08.362558 5451 log.cc:824] T 00000000000000000000000000000000 P 38a4776bf9a24bdd955b1db12eb0034b: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:08.363334 5451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38a4776bf9a24bdd955b1db12eb0034b: No bootstrap required, opened a new log
I20260925 19:05:08.363919 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 5384
I20260925 19:05:08.364018 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:05:08.364198 787 external_mini_cluster.cc:1428] Setting key e164031333040f7bac91c459bd6bb445
I20260925 19:05:08.365528 5451 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38a4776bf9a24bdd955b1db12eb0034b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a4776bf9a24bdd955b1db12eb0034b" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 42701 } }
I20260925 19:05:08.365696 5451 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38a4776bf9a24bdd955b1db12eb0034b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:08.365736 5451 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38a4776bf9a24bdd955b1db12eb0034b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38a4776bf9a24bdd955b1db12eb0034b, State: Initialized, Role: FOLLOWER
I20260925 19:05:08.365899 5451 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38a4776bf9a24bdd955b1db12eb0034b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a4776bf9a24bdd955b1db12eb0034b" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 42701 } }
I20260925 19:05:08.366003 5451 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38a4776bf9a24bdd955b1db12eb0034b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:05:08.366042 5451 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38a4776bf9a24bdd955b1db12eb0034b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:05:08.366092 5451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38a4776bf9a24bdd955b1db12eb0034b [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:08.366807 5451 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38a4776bf9a24bdd955b1db12eb0034b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a4776bf9a24bdd955b1db12eb0034b" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 42701 } }
I20260925 19:05:08.366974 5451 leader_election.cc:302] T 00000000000000000000000000000000 P 38a4776bf9a24bdd955b1db12eb0034b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38a4776bf9a24bdd955b1db12eb0034b; no voters:
I20260925 19:05:08.367026 5455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:08.364800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53206 (local address 127.0.196.254:42701)
0925 19:05:08.365049 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:08.365058 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:08.365086 (+ 28us) server_negotiation.cc:407] Connection header received
0925 19:05:08.365509 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:08.365515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:08.365569 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:08.365685 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:08.366432 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:08.366446 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:08.366446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:05:08.366451 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:08.366496 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:05:08.366514 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:08.366602 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:08.366738 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 19:05:08.366801 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":127,"threads_started":1}
I20260925 19:05:08.367141 5451 leader_election.cc:288] T 00000000000000000000000000000000 P 38a4776bf9a24bdd955b1db12eb0034b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:05:08.367401 5456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38a4776bf9a24bdd955b1db12eb0034b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:05:08.367416 5451 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38a4776bf9a24bdd955b1db12eb0034b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:05:08.367645 5456 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38a4776bf9a24bdd955b1db12eb0034b [term 1 LEADER]: Becoming Leader. State: Replica: 38a4776bf9a24bdd955b1db12eb0034b, State: Running, Role: LEADER
I20260925 19:05:08.367761 5456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38a4776bf9a24bdd955b1db12eb0034b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a4776bf9a24bdd955b1db12eb0034b" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 42701 } }
I20260925 19:05:08.368139 5457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38a4776bf9a24bdd955b1db12eb0034b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38a4776bf9a24bdd955b1db12eb0034b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a4776bf9a24bdd955b1db12eb0034b" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 42701 } } }
I20260925 19:05:08.368155 5458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38a4776bf9a24bdd955b1db12eb0034b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38a4776bf9a24bdd955b1db12eb0034b. Latest consensus state: current_term: 1 leader_uuid: "38a4776bf9a24bdd955b1db12eb0034b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a4776bf9a24bdd955b1db12eb0034b" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 42701 } } }
I20260925 19:05:08.368237 5457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38a4776bf9a24bdd955b1db12eb0034b [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:08.368263 5458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38a4776bf9a24bdd955b1db12eb0034b [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:08.368446 5465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:05:08.369021 5465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:05:08.370514 5465 catalog_manager.cc:1347] Generated new cluster ID: 122ecaa3a55746edb45e5e10fa18f170
I20260925 19:05:08.370584 5465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:05:08.386200 5465 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:05:08.386404 5465 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:05:08.391036 5465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38a4776bf9a24bdd955b1db12eb0034b: Generated new TSK 0
I20260925 19:05:08.391292 5465 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:05:08.401232 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:0
--local_ip_for_outbound_sockets=127.0.196.193
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:42701
--builtin_ntp_servers=127.0.196.212:33091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 19:05:08.499274 5475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:08.499533 5475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:08.499576 5475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:08.500890 5475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:08.500962 5475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:08.501008 5475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:05:08.502447 5475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:33091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:42701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:08.502785 5475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:08.503109 5475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:08.508368 5481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:08.508330 5483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:08.508338 5480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:08.509301 5475 server_base.cc:1034] running on GCE node
I20260925 19:05:08.509579 5475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:08.509851 5475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:08.511023 5475 hybrid_clock.cc:648] HybridClock initialized: now 1790363108511010 us; error 33 us; skew 500 ppm
I20260925 19:05:08.512446 5475 webserver.cc:460] Webserver started at http://127.0.196.193:36589/ using document root <none> and password file <none>
I20260925 19:05:08.512723 5475 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:08.512794 5475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:08.512944 5475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:08.514051 5475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-0/data/instance:
uuid: "55662971924f430daff6aea93acfafa8"
format_stamp: "Formatted at 2026-09-25 19:05:08 on dist-test-slave-rn98"
server_key: "6eb615658020a681e7667bdf6afe944f"
server_key_iv: "024eef8fe39d153d335d1b2f3d42a4f9"
server_key_version: "encryptionkey@0"
I20260925 19:05:08.514438 5475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-0/wal/instance:
uuid: "55662971924f430daff6aea93acfafa8"
format_stamp: "Formatted at 2026-09-25 19:05:08 on dist-test-slave-rn98"
server_key: "6eb615658020a681e7667bdf6afe944f"
server_key_iv: "024eef8fe39d153d335d1b2f3d42a4f9"
server_key_version: "encryptionkey@0"
I20260925 19:05:08.515945 5475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:05:08.516853 5489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:08.517081 5475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:05:08.517200 5475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "55662971924f430daff6aea93acfafa8"
format_stamp: "Formatted at 2026-09-25 19:05:08 on dist-test-slave-rn98"
server_key: "6eb615658020a681e7667bdf6afe944f"
server_key_iv: "024eef8fe39d153d335d1b2f3d42a4f9"
server_key_version: "encryptionkey@0"
I20260925 19:05:08.517279 5475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:08.522921 5475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:08.523279 5475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:08.523432 5475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:08.523674 5475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:08.524044 5475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:08.524091 5475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:08.524121 5475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:08.524147 5475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:08.531132 5475 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:43527
I20260925 19:05:08.531255 5602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:43527 every 8 connection(s)
I20260925 19:05:08.531648 5475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
I20260925 19:05:08.535063 5455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:08.532533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:39749 (local address 127.0.196.254:42701)
0925 19:05:08.532593 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:08.532596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:08.532825 (+ 229us) server_negotiation.cc:407] Connection header received
0925 19:05:08.533405 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:08.533407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:08.533415 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:08.533461 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:08.534779 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:08.534781 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:08.534781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:05:08.534786 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:08.534803 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:05:08.534808 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:08.534868 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:08.534969 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 19:05:08.534987 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 19:05:08.535296 5605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:08.532501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:42701 (local address 127.0.196.193:39749)
0925 19:05:08.532693 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:08.532728 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:08.533242 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:08.533551 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:08.533555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:08.533563 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:08.534572 (+ 1009us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:05:08.534573 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:08.534642 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:05:08.534675 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:08.534876 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:08.534882 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:08.534886 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:05:08.534981 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 19:05:08.535046 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":88,"threads_started":1}
I20260925 19:05:08.535851 5603 heartbeater.cc:347] Connected to a master server at 127.0.196.254:42701
I20260925 19:05:08.535935 5603 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:08.536063 5603 heartbeater.cc:511] Master 127.0.196.254:42701 requested a full tablet report, sending...
I20260925 19:05:08.536546 5415 ts_manager.cc:194] Registered new tserver with Master: 55662971924f430daff6aea93acfafa8 (127.0.196.193:43527)
I20260925 19:05:08.537027 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 5475
I20260925 19:05:08.537149 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-0/wal/instance
I20260925 19:05:08.537341 787 external_mini_cluster.cc:1428] Setting key 449c3f4faa0a8cabcd4c51f540d4be65
I20260925 19:05:08.538240 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:0
--local_ip_for_outbound_sockets=127.0.196.194
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:42701
--builtin_ntp_servers=127.0.196.212:33091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 19:05:08.621706 5606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:08.622009 5606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:08.622061 5606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:08.623476 5606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:08.623525 5606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:08.623553 5606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:05:08.624972 5606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:33091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:42701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:08.625260 5606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:08.625605 5606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:08.631192 5612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:08.631196 5614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:08.631192 5611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:08.631301 5606 server_base.cc:1034] running on GCE node
I20260925 19:05:08.631695 5606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:08.631956 5606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:08.633136 5606 hybrid_clock.cc:648] HybridClock initialized: now 1790363108633132 us; error 48 us; skew 500 ppm
I20260925 19:05:08.639957 5606 webserver.cc:460] Webserver started at http://127.0.196.194:33117/ using document root <none> and password file <none>
I20260925 19:05:08.640275 5606 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:08.640355 5606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:08.640518 5606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:08.641731 5606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-1/data/instance:
uuid: "733ff9fb4adb470d8b98f1cda4c34745"
format_stamp: "Formatted at 2026-09-25 19:05:08 on dist-test-slave-rn98"
server_key: "b12ca7ec66e9f6ba0747988e6ea4fbb1"
server_key_iv: "9a50263c4399c6510f2de0fbf0a81c0d"
server_key_version: "encryptionkey@0"
I20260925 19:05:08.642131 5606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-1/wal/instance:
uuid: "733ff9fb4adb470d8b98f1cda4c34745"
format_stamp: "Formatted at 2026-09-25 19:05:08 on dist-test-slave-rn98"
server_key: "b12ca7ec66e9f6ba0747988e6ea4fbb1"
server_key_iv: "9a50263c4399c6510f2de0fbf0a81c0d"
server_key_version: "encryptionkey@0"
I20260925 19:05:08.643585 5606 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:05:08.644459 5620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:08.644658 5606 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:08.644776 5606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "733ff9fb4adb470d8b98f1cda4c34745"
format_stamp: "Formatted at 2026-09-25 19:05:08 on dist-test-slave-rn98"
server_key: "b12ca7ec66e9f6ba0747988e6ea4fbb1"
server_key_iv: "9a50263c4399c6510f2de0fbf0a81c0d"
server_key_version: "encryptionkey@0"
I20260925 19:05:08.644861 5606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:08.648381 5606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:08.648679 5606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:08.648823 5606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:08.649039 5606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:08.649358 5606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:08.649411 5606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:08.649451 5606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:08.649478 5606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:08.655764 5606 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:44569
I20260925 19:05:08.655805 5733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:44569 every 8 connection(s)
I20260925 19:05:08.656172 5606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
I20260925 19:05:08.659540 5455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:08.656922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:41027 (local address 127.0.196.254:42701)
0925 19:05:08.656971 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:08.656973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:08.657224 (+ 251us) server_negotiation.cc:407] Connection header received
0925 19:05:08.657707 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:08.657708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:08.657714 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:08.657772 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:08.659295 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:08.659297 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:08.659297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:05:08.659301 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:08.659319 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:05:08.659324 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:08.659376 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:08.659470 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 19:05:08.659483 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:05:08.659793 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:08.656922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:42701 (local address 127.0.196.194:41027)
0925 19:05:08.657096 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:08.657130 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:08.657590 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:08.657827 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:08.657831 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:08.657838 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:08.658937 (+ 1099us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:05:08.658938 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:08.658967 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:05:08.658998 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:08.659382 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:08.659388 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:08.659391 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:05:08.659471 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 19:05:08.659532 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260925 19:05:08.660431 5734 heartbeater.cc:347] Connected to a master server at 127.0.196.254:42701
I20260925 19:05:08.660532 5734 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:08.660854 5734 heartbeater.cc:511] Master 127.0.196.254:42701 requested a full tablet report, sending...
I20260925 19:05:08.661341 5415 ts_manager.cc:194] Registered new tserver with Master: 733ff9fb4adb470d8b98f1cda4c34745 (127.0.196.194:44569)
I20260925 19:05:08.663323 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 5606
I20260925 19:05:08.663410 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-1/wal/instance
I20260925 19:05:08.663547 787 external_mini_cluster.cc:1428] Setting key 9b068dc64cc3dc902d6db2a4448ed19b
I20260925 19:05:08.664559 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:0
--local_ip_for_outbound_sockets=127.0.196.195
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:42701
--builtin_ntp_servers=127.0.196.212:33091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 19:05:08.750013 5737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:08.750245 5737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:08.750269 5737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:08.751566 5737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:08.751612 5737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:08.751641 5737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:05:08.753243 5737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:33091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:42701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:08.753520 5737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:08.753829 5737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:08.759069 5745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:08.759109 5742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:08.759347 5743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:08.759372 5737 server_base.cc:1034] running on GCE node
I20260925 19:05:08.759609 5737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:08.759865 5737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:08.761031 5737 hybrid_clock.cc:648] HybridClock initialized: now 1790363108761020 us; error 60 us; skew 500 ppm
I20260925 19:05:08.762458 5737 webserver.cc:460] Webserver started at http://127.0.196.195:45461/ using document root <none> and password file <none>
I20260925 19:05:08.762765 5737 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:08.762838 5737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:08.763000 5737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:08.764496 5737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-2/data/instance:
uuid: "43622c34489e404999d0916e6bcb58aa"
format_stamp: "Formatted at 2026-09-25 19:05:08 on dist-test-slave-rn98"
server_key: "3ffaec1eb3aed3e6ac0027e4b78793e2"
server_key_iv: "db6745f37906309ceb0164d67f006fef"
server_key_version: "encryptionkey@0"
I20260925 19:05:08.764884 5737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-2/wal/instance:
uuid: "43622c34489e404999d0916e6bcb58aa"
format_stamp: "Formatted at 2026-09-25 19:05:08 on dist-test-slave-rn98"
server_key: "3ffaec1eb3aed3e6ac0027e4b78793e2"
server_key_iv: "db6745f37906309ceb0164d67f006fef"
server_key_version: "encryptionkey@0"
I20260925 19:05:08.766291 5737 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:05:08.767100 5751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:08.767329 5737 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:05:08.767421 5737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "43622c34489e404999d0916e6bcb58aa"
format_stamp: "Formatted at 2026-09-25 19:05:08 on dist-test-slave-rn98"
server_key: "3ffaec1eb3aed3e6ac0027e4b78793e2"
server_key_iv: "db6745f37906309ceb0164d67f006fef"
server_key_version: "encryptionkey@0"
I20260925 19:05:08.767498 5737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:08.771034 5737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:08.771314 5737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:08.771458 5737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:08.771667 5737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:08.771971 5737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:08.772025 5737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:08.772066 5737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:08.772099 5737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:08.777770 5737 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:33599
I20260925 19:05:08.777812 5864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:33599 every 8 connection(s)
I20260925 19:05:08.778162 5737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
I20260925 19:05:08.779778 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 5737
I20260925 19:05:08.779888 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-2/wal/instance
I20260925 19:05:08.780120 787 external_mini_cluster.cc:1428] Setting key 15d0c6349984f9cc862a0dce9dadb9c8
I20260925 19:05:08.781435 5455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:08.778942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:35727 (local address 127.0.196.254:42701)
0925 19:05:08.779005 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:08.779007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:08.779248 (+ 241us) server_negotiation.cc:407] Connection header received
0925 19:05:08.779739 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:08.779740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:08.779748 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:08.779826 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:08.781252 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:08.781253 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:08.781254 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:05:08.781257 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:08.781272 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:05:08.781275 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:08.781322 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:08.781369 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 19:05:08.781381 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 19:05:08.781672 5867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:08.778930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:42701 (local address 127.0.196.195:35727)
0925 19:05:08.779109 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:08.779154 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:08.779626 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:08.779928 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:08.779933 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:08.779944 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:08.781113 (+ 1169us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:05:08.781114 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:08.781145 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:05:08.781175 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:08.781320 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:08.781323 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:08.781325 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:05:08.781375 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 19:05:08.781423 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260925 19:05:08.782151 5865 heartbeater.cc:347] Connected to a master server at 127.0.196.254:42701
I20260925 19:05:08.782258 5865 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:08.782408 5865 heartbeater.cc:511] Master 127.0.196.254:42701 requested a full tablet report, sending...
I20260925 19:05:08.782826 5415 ts_manager.cc:194] Registered new tserver with Master: 43622c34489e404999d0916e6bcb58aa (127.0.196.195:33599)
I20260925 19:05:08.792121 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:05:08.794943 5455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:08.793496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53216 (local address 127.0.196.254:42701)
0925 19:05:08.793673 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:08.793675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:08.793776 (+ 101us) server_negotiation.cc:407] Connection header received
0925 19:05:08.793873 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:08.793874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:08.793880 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:08.793953 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:08.794712 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:08.794713 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:08.794714 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:05:08.794718 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:08.794738 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:05:08.794742 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:08.794788 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:08.794850 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 19:05:08.794865 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260925 19:05:08.796804 5415 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:05:08.798692 5415 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:05:08.802986 5877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:08.801558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48842 (local address 127.0.196.194:44569)
0925 19:05:08.801744 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:08.801749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:08.801760 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:05:08.801788 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:08.801791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:08.801800 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:08.802165 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:08.802669 (+ 504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:08.802677 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:08.802677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:05:08.802682 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:08.802719 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:05:08.802748 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:08.802817 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:08.802891 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 19:05:08.802905 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":113,"threads_started":1}
I20260925 19:05:08.802975 5875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:08.801096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:44569 (local address 127.0.0.1:48842)
0925 19:05:08.801352 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:08.801362 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:08.801408 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:08.801910 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:08.801917 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:08.801926 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:08.802497 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:05:08.802497 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:08.802515 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:05:08.802542 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:08.802813 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:08.802819 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:08.802823 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:05:08.802886 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 19:05:08.802900 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":101,"threads_started":1}
I20260925 19:05:08.803133 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:08.801096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:43527 (local address 127.0.0.1:52214)
0925 19:05:08.801490 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:08.801499 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:08.801544 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:08.801957 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:08.801957 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:08.801959 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:08.802684 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:05:08.802685 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:08.802691 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:05:08.802703 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:08.802915 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:08.802916 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:08.802917 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 19:05:08.802968 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 19:05:08.802982 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":3,"thread_start_us":104,"threads_started":1}
I20260925 19:05:08.803345 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:08.801571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52214 (local address 127.0.196.193:43527)
0925 19:05:08.801793 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:08.801796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:08.801805 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:05:08.801829 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:08.801831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:08.801840 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:08.802180 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:08.802782 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:08.802789 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:08.802789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:05:08.802794 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:08.802833 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:05:08.802856 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:08.802924 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:08.803021 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 19:05:08.803241 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":100,"threads_started":1}
I20260925 19:05:08.803380 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:08.801039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:33599 (local address 127.0.0.1:36732)
0925 19:05:08.801173 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:08.801213 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:08.801270 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:08.801929 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:08.801930 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:08.801932 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:08.802977 (+ 1045us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:05:08.802978 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:08.802984 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:05:08.802994 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:08.803250 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:08.803251 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:08.803252 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 19:05:08.803299 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 19:05:08.803310 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":87,"threads_started":1}
I20260925 19:05:08.803421 5876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:08.801275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36732 (local address 127.0.196.195:33599)
0925 19:05:08.801452 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:08.801456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:08.801464 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:05:08.801504 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:08.801506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:08.801516 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:08.801590 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:08.803088 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:08.803097 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:08.803097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:05:08.803102 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:08.803142 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:05:08.803169 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:08.803227 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:08.803315 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 19:05:08.803331 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":91,"threads_started":1}
I20260925 19:05:08.804077 5799 tablet_service.cc:1467] Processing CreateTablet for tablet 41e0565c34a74925916fd7e5170b4602 (DEFAULT_TABLE table=test-table [id=a3d8f9edbaf84bbd94691c4b16fde0ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:05:08.804081 5668 tablet_service.cc:1467] Processing CreateTablet for tablet 41e0565c34a74925916fd7e5170b4602 (DEFAULT_TABLE table=test-table [id=a3d8f9edbaf84bbd94691c4b16fde0ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:05:08.804450 5668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41e0565c34a74925916fd7e5170b4602. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:08.804464 5799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41e0565c34a74925916fd7e5170b4602. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:08.807088 5537 tablet_service.cc:1467] Processing CreateTablet for tablet 41e0565c34a74925916fd7e5170b4602 (DEFAULT_TABLE table=test-table [id=a3d8f9edbaf84bbd94691c4b16fde0ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:05:08.807216 5879 tablet_bootstrap.cc:492] T 41e0565c34a74925916fd7e5170b4602 P 43622c34489e404999d0916e6bcb58aa: Bootstrap starting.
I20260925 19:05:08.807442 5537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41e0565c34a74925916fd7e5170b4602. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:08.807574 5880 tablet_bootstrap.cc:492] T 41e0565c34a74925916fd7e5170b4602 P 733ff9fb4adb470d8b98f1cda4c34745: Bootstrap starting.
I20260925 19:05:08.807791 5879 tablet_bootstrap.cc:654] T 41e0565c34a74925916fd7e5170b4602 P 43622c34489e404999d0916e6bcb58aa: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:08.808012 5879 log.cc:824] T 41e0565c34a74925916fd7e5170b4602 P 43622c34489e404999d0916e6bcb58aa: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:08.808197 5880 tablet_bootstrap.cc:654] T 41e0565c34a74925916fd7e5170b4602 P 733ff9fb4adb470d8b98f1cda4c34745: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:08.808487 5880 log.cc:824] T 41e0565c34a74925916fd7e5170b4602 P 733ff9fb4adb470d8b98f1cda4c34745: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:08.808687 5879 tablet_bootstrap.cc:492] T 41e0565c34a74925916fd7e5170b4602 P 43622c34489e404999d0916e6bcb58aa: No bootstrap required, opened a new log
I20260925 19:05:08.808751 5879 ts_tablet_manager.cc:1397] T 41e0565c34a74925916fd7e5170b4602 P 43622c34489e404999d0916e6bcb58aa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:05:08.809257 5880 tablet_bootstrap.cc:492] T 41e0565c34a74925916fd7e5170b4602 P 733ff9fb4adb470d8b98f1cda4c34745: No bootstrap required, opened a new log
I20260925 19:05:08.809314 5880 ts_tablet_manager.cc:1397] T 41e0565c34a74925916fd7e5170b4602 P 733ff9fb4adb470d8b98f1cda4c34745: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:05:08.810909 5883 tablet_bootstrap.cc:492] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8: Bootstrap starting.
I20260925 19:05:08.811335 5879 raft_consensus.cc:348] T 41e0565c34a74925916fd7e5170b4602 P 43622c34489e404999d0916e6bcb58aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43622c34489e404999d0916e6bcb58aa" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33599 } } peers { permanent_uuid: "55662971924f430daff6aea93acfafa8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 43527 } } peers { permanent_uuid: "733ff9fb4adb470d8b98f1cda4c34745" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44569 } }
I20260925 19:05:08.811533 5879 raft_consensus.cc:374] T 41e0565c34a74925916fd7e5170b4602 P 43622c34489e404999d0916e6bcb58aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:08.811578 5879 raft_consensus.cc:729] T 41e0565c34a74925916fd7e5170b4602 P 43622c34489e404999d0916e6bcb58aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43622c34489e404999d0916e6bcb58aa, State: Initialized, Role: FOLLOWER
I20260925 19:05:08.811594 5883 tablet_bootstrap.cc:654] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:08.811738 5879 consensus_queue.cc:260] T 41e0565c34a74925916fd7e5170b4602 P 43622c34489e404999d0916e6bcb58aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43622c34489e404999d0916e6bcb58aa" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33599 } } peers { permanent_uuid: "55662971924f430daff6aea93acfafa8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 43527 } } peers { permanent_uuid: "733ff9fb4adb470d8b98f1cda4c34745" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44569 } }
I20260925 19:05:08.811911 5883 log.cc:824] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:08.811914 5880 raft_consensus.cc:348] T 41e0565c34a74925916fd7e5170b4602 P 733ff9fb4adb470d8b98f1cda4c34745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43622c34489e404999d0916e6bcb58aa" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33599 } } peers { permanent_uuid: "55662971924f430daff6aea93acfafa8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 43527 } } peers { permanent_uuid: "733ff9fb4adb470d8b98f1cda4c34745" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44569 } }
I20260925 19:05:08.812031 5879 ts_tablet_manager.cc:1428] T 41e0565c34a74925916fd7e5170b4602 P 43622c34489e404999d0916e6bcb58aa: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 19:05:08.812103 5880 raft_consensus.cc:374] T 41e0565c34a74925916fd7e5170b4602 P 733ff9fb4adb470d8b98f1cda4c34745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:08.812147 5880 raft_consensus.cc:729] T 41e0565c34a74925916fd7e5170b4602 P 733ff9fb4adb470d8b98f1cda4c34745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 733ff9fb4adb470d8b98f1cda4c34745, State: Initialized, Role: FOLLOWER
I20260925 19:05:08.812163 5865 heartbeater.cc:503] Master 127.0.196.254:42701 was elected leader, sending a full tablet report...
I20260925 19:05:08.812253 5880 consensus_queue.cc:260] T 41e0565c34a74925916fd7e5170b4602 P 733ff9fb4adb470d8b98f1cda4c34745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43622c34489e404999d0916e6bcb58aa" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33599 } } peers { permanent_uuid: "55662971924f430daff6aea93acfafa8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 43527 } } peers { permanent_uuid: "733ff9fb4adb470d8b98f1cda4c34745" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44569 } }
I20260925 19:05:08.812510 5880 ts_tablet_manager.cc:1428] T 41e0565c34a74925916fd7e5170b4602 P 733ff9fb4adb470d8b98f1cda4c34745: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 19:05:08.812541 5734 heartbeater.cc:503] Master 127.0.196.254:42701 was elected leader, sending a full tablet report...
I20260925 19:05:08.813257 5883 tablet_bootstrap.cc:492] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8: No bootstrap required, opened a new log
I20260925 19:05:08.813329 5883 ts_tablet_manager.cc:1397] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:05:08.815244 5883 raft_consensus.cc:348] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43622c34489e404999d0916e6bcb58aa" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33599 } } peers { permanent_uuid: "55662971924f430daff6aea93acfafa8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 43527 } } peers { permanent_uuid: "733ff9fb4adb470d8b98f1cda4c34745" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44569 } }
I20260925 19:05:08.815371 5883 raft_consensus.cc:374] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:08.815392 5883 raft_consensus.cc:729] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55662971924f430daff6aea93acfafa8, State: Initialized, Role: FOLLOWER
I20260925 19:05:08.815501 5883 consensus_queue.cc:260] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43622c34489e404999d0916e6bcb58aa" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33599 } } peers { permanent_uuid: "55662971924f430daff6aea93acfafa8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 43527 } } peers { permanent_uuid: "733ff9fb4adb470d8b98f1cda4c34745" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44569 } }
I20260925 19:05:08.815733 5883 ts_tablet_manager.cc:1428] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 19:05:08.815801 5603 heartbeater.cc:503] Master 127.0.196.254:42701 was elected leader, sending a full tablet report...
W20260925 19:05:08.906811 5735 tablet.cc:2369] T 41e0565c34a74925916fd7e5170b4602 P 733ff9fb4adb470d8b98f1cda4c34745: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:05:09.028728 5866 tablet.cc:2369] T 41e0565c34a74925916fd7e5170b4602 P 43622c34489e404999d0916e6bcb58aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:05:09.032436 5604 tablet.cc:2369] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:05:09.124714 5887 raft_consensus.cc:482] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:05:09.124890 5887 raft_consensus.cc:504] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43622c34489e404999d0916e6bcb58aa" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33599 } } peers { permanent_uuid: "55662971924f430daff6aea93acfafa8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 43527 } } peers { permanent_uuid: "733ff9fb4adb470d8b98f1cda4c34745" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44569 } }
I20260925 19:05:09.125278 5887 leader_election.cc:288] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43622c34489e404999d0916e6bcb58aa (127.0.196.195:33599), 733ff9fb4adb470d8b98f1cda4c34745 (127.0.196.194:44569)
I20260925 19:05:09.126878 5876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:09.125645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:51979 (local address 127.0.196.195:33599)
0925 19:05:09.125692 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:09.125696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:09.125728 (+ 32us) server_negotiation.cc:407] Connection header received
0925 19:05:09.125805 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:09.125806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:09.125810 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:09.125862 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:09.126646 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:09.126648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:09.126649 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:05:09.126653 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:09.126668 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:05:09.126673 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:09.126738 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:09.126792 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 19:05:09.126806 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 19:05:09.126878 5877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:09.125590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:39869 (local address 127.0.196.194:44569)
0925 19:05:09.125642 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:09.125644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:09.125729 (+ 85us) server_negotiation.cc:407] Connection header received
0925 19:05:09.125786 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:09.125787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:09.125794 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:09.125862 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:09.126554 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:09.126555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:09.126555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:05:09.126559 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:09.126608 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:05:09.126641 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:09.126706 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:09.126790 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 19:05:09.126806 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 19:05:09.126907 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:09.125516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:44569 (local address 127.0.196.193:39869)
0925 19:05:09.125644 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:09.125657 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:09.125722 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:09.125918 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:09.125920 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:09.125924 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:09.126405 (+ 481us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:05:09.126405 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:09.126412 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:05:09.126453 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:09.126721 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:09.126721 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:09.126722 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 19:05:09.126780 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 19:05:09.126797 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260925 19:05:09.126907 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:09.125516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:33599 (local address 127.0.196.193:51979)
0925 19:05:09.125644 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:09.125657 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:09.125722 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:09.125918 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:09.125920 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:09.125924 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:09.126518 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:05:09.126518 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:09.126524 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:05:09.126535 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:09.126737 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:09.126737 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:09.126738 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 19:05:09.126782 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 19:05:09.126797 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":2,"thread_start_us":61,"threads_started":1}
I20260925 19:05:09.127394 5688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e0565c34a74925916fd7e5170b4602" candidate_uuid: "55662971924f430daff6aea93acfafa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "733ff9fb4adb470d8b98f1cda4c34745" is_pre_election: true
I20260925 19:05:09.127452 5819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e0565c34a74925916fd7e5170b4602" candidate_uuid: "55662971924f430daff6aea93acfafa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43622c34489e404999d0916e6bcb58aa" is_pre_election: true
I20260925 19:05:09.127583 5688 raft_consensus.cc:2454] T 41e0565c34a74925916fd7e5170b4602 P 733ff9fb4adb470d8b98f1cda4c34745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55662971924f430daff6aea93acfafa8 in term 0.
I20260925 19:05:09.127640 5819 raft_consensus.cc:2454] T 41e0565c34a74925916fd7e5170b4602 P 43622c34489e404999d0916e6bcb58aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55662971924f430daff6aea93acfafa8 in term 0.
I20260925 19:05:09.127880 5491 leader_election.cc:302] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 43622c34489e404999d0916e6bcb58aa, 55662971924f430daff6aea93acfafa8, 733ff9fb4adb470d8b98f1cda4c34745; no voters:
I20260925 19:05:09.128002 5887 raft_consensus.cc:2781] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:05:09.128054 5887 raft_consensus.cc:482] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:05:09.128077 5887 raft_consensus.cc:3037] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:09.128778 5887 raft_consensus.cc:504] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43622c34489e404999d0916e6bcb58aa" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33599 } } peers { permanent_uuid: "55662971924f430daff6aea93acfafa8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 43527 } } peers { permanent_uuid: "733ff9fb4adb470d8b98f1cda4c34745" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44569 } }
I20260925 19:05:09.128921 5887 leader_election.cc:288] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8 [CANDIDATE]: Term 1 election: Requested vote from peers 43622c34489e404999d0916e6bcb58aa (127.0.196.195:33599), 733ff9fb4adb470d8b98f1cda4c34745 (127.0.196.194:44569)
I20260925 19:05:09.129067 5688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e0565c34a74925916fd7e5170b4602" candidate_uuid: "55662971924f430daff6aea93acfafa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "733ff9fb4adb470d8b98f1cda4c34745"
I20260925 19:05:09.129082 5819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e0565c34a74925916fd7e5170b4602" candidate_uuid: "55662971924f430daff6aea93acfafa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43622c34489e404999d0916e6bcb58aa"
I20260925 19:05:09.129139 5688 raft_consensus.cc:3037] T 41e0565c34a74925916fd7e5170b4602 P 733ff9fb4adb470d8b98f1cda4c34745 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:09.129184 5819 raft_consensus.cc:3037] T 41e0565c34a74925916fd7e5170b4602 P 43622c34489e404999d0916e6bcb58aa [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:09.129842 5688 raft_consensus.cc:2454] T 41e0565c34a74925916fd7e5170b4602 P 733ff9fb4adb470d8b98f1cda4c34745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55662971924f430daff6aea93acfafa8 in term 1.
I20260925 19:05:09.129938 5819 raft_consensus.cc:2454] T 41e0565c34a74925916fd7e5170b4602 P 43622c34489e404999d0916e6bcb58aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55662971924f430daff6aea93acfafa8 in term 1.
I20260925 19:05:09.129998 5491 leader_election.cc:302] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55662971924f430daff6aea93acfafa8, 733ff9fb4adb470d8b98f1cda4c34745; no voters:
I20260925 19:05:09.130118 5887 raft_consensus.cc:2781] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:05:09.130280 5887 raft_consensus.cc:686] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8 [term 1 LEADER]: Becoming Leader. State: Replica: 55662971924f430daff6aea93acfafa8, State: Running, Role: LEADER
I20260925 19:05:09.130393 5887 consensus_queue.cc:237] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43622c34489e404999d0916e6bcb58aa" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33599 } } peers { permanent_uuid: "55662971924f430daff6aea93acfafa8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 43527 } } peers { permanent_uuid: "733ff9fb4adb470d8b98f1cda4c34745" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44569 } }
I20260925 19:05:09.131256 5415 catalog_manager.cc:5495] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55662971924f430daff6aea93acfafa8 (127.0.196.193). New cstate: current_term: 1 leader_uuid: "55662971924f430daff6aea93acfafa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43622c34489e404999d0916e6bcb58aa" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55662971924f430daff6aea93acfafa8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "733ff9fb4adb470d8b98f1cda4c34745" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44569 } health_report { overall_health: UNKNOWN } } }
I20260925 19:05:09.157861 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:09.156896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52226 (local address 127.0.196.193:43527)
0925 19:05:09.156946 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:09.156948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:09.156956 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:05:09.157000 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:09.157000 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:09.157015 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:09.157076 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:09.157685 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:09.157686 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:09.157687 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:05:09.157691 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:09.157706 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:05:09.157710 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:09.157763 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:09.157795 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 19:05:09.157806 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:05:09.159878 5688 raft_consensus.cc:1261] T 41e0565c34a74925916fd7e5170b4602 P 733ff9fb4adb470d8b98f1cda4c34745 [term 1 FOLLOWER]: Refusing update from remote peer 55662971924f430daff6aea93acfafa8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:05:09.159904 5819 raft_consensus.cc:1261] T 41e0565c34a74925916fd7e5170b4602 P 43622c34489e404999d0916e6bcb58aa [term 1 FOLLOWER]: Refusing update from remote peer 55662971924f430daff6aea93acfafa8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:05:09.160290 5887 consensus_queue.cc:1035] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "733ff9fb4adb470d8b98f1cda4c34745" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 44569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:05:09.160451 5890 consensus_queue.cc:1035] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43622c34489e404999d0916e6bcb58aa" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 33599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:05:09.162735 5893 mvcc.cc:205] Tried to move back new op lower bound from 7333327295116218368 to 7333327294998839296. Current Snapshot: MvccSnapshot[applied={T|T < 7333327295116218368}]
I20260925 19:05:09.163048 5894 mvcc.cc:205] Tried to move back new op lower bound from 7333327295116218368 to 7333327294998839296. Current Snapshot: MvccSnapshot[applied={T|T < 7333327295116218368}]
I20260925 19:05:09.163681 5895 mvcc.cc:205] Tried to move back new op lower bound from 7333327295116218368 to 7333327294998839296. Current Snapshot: MvccSnapshot[applied={T|T < 7333327295116218368}]
I20260925 19:05:09.165336 5414 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:05:09.165441 5414 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:05:09.166226 5414 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a4776bf9a24bdd955b1db12eb0034b: Sending DeleteTablet for 3 replicas of tablet 41e0565c34a74925916fd7e5170b4602
I20260925 19:05:09.166647 5537 tablet_service.cc:1514] Processing DeleteTablet for tablet 41e0565c34a74925916fd7e5170b4602 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:05:09 UTC) from {username='slave'} at 127.0.0.1:52214
I20260925 19:05:09.166689 5799 tablet_service.cc:1514] Processing DeleteTablet for tablet 41e0565c34a74925916fd7e5170b4602 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:05:09 UTC) from {username='slave'} at 127.0.0.1:36732
I20260925 19:05:09.166711 5668 tablet_service.cc:1514] Processing DeleteTablet for tablet 41e0565c34a74925916fd7e5170b4602 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:05:09 UTC) from {username='slave'} at 127.0.0.1:48842
I20260925 19:05:09.166910 5904 tablet_replica.cc:330] T 41e0565c34a74925916fd7e5170b4602 P 733ff9fb4adb470d8b98f1cda4c34745: stopping tablet replica
I20260925 19:05:09.167019 5904 raft_consensus.cc:2229] T 41e0565c34a74925916fd7e5170b4602 P 733ff9fb4adb470d8b98f1cda4c34745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:05:09.167019 5903 tablet_replica.cc:330] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8: stopping tablet replica
I20260925 19:05:09.167124 5904 raft_consensus.cc:2258] T 41e0565c34a74925916fd7e5170b4602 P 733ff9fb4adb470d8b98f1cda4c34745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:05:09.167128 5903 raft_consensus.cc:2229] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:05:09.167220 5903 raft_consensus.cc:2258] T 41e0565c34a74925916fd7e5170b4602 P 55662971924f430daff6aea93acfafa8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:05:09.167274 5905 tablet_replica.cc:330] T 41e0565c34a74925916fd7e5170b4602 P 43622c34489e404999d0916e6bcb58aa: stopping tablet replica
I20260925 19:05:09.167459 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 5475
I20260925 19:05:09.167829 5904 ts_tablet_manager.cc:1905] T 41e0565c34a74925916fd7e5170b4602 P 733ff9fb4adb470d8b98f1cda4c34745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:05:09.168006 5905 raft_consensus.cc:2229] T 41e0565c34a74925916fd7e5170b4602 P 43622c34489e404999d0916e6bcb58aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:05:09.168114 5905 raft_consensus.cc:2258] T 41e0565c34a74925916fd7e5170b4602 P 43622c34489e404999d0916e6bcb58aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:05:09.169433 5905 ts_tablet_manager.cc:1905] T 41e0565c34a74925916fd7e5170b4602 P 43622c34489e404999d0916e6bcb58aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:05:09.169975 5904 ts_tablet_manager.cc:1918] T 41e0565c34a74925916fd7e5170b4602 P 733ff9fb4adb470d8b98f1cda4c34745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:05:09.170051 5904 log.cc:1196] T 41e0565c34a74925916fd7e5170b4602 P 733ff9fb4adb470d8b98f1cda4c34745: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-1/wal/wals/41e0565c34a74925916fd7e5170b4602
I20260925 19:05:09.170327 5904 ts_tablet_manager.cc:1939] T 41e0565c34a74925916fd7e5170b4602 P 733ff9fb4adb470d8b98f1cda4c34745: Deleting consensus metadata
I20260925 19:05:09.170840 5400 catalog_manager.cc:4841] TS 733ff9fb4adb470d8b98f1cda4c34745 (127.0.196.194:44569): tablet 41e0565c34a74925916fd7e5170b4602 (table test-table [id=a3d8f9edbaf84bbd94691c4b16fde0ce]) successfully deleted
I20260925 19:05:09.171185 5905 ts_tablet_manager.cc:1918] T 41e0565c34a74925916fd7e5170b4602 P 43622c34489e404999d0916e6bcb58aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:05:09.171269 5905 log.cc:1196] T 41e0565c34a74925916fd7e5170b4602 P 43622c34489e404999d0916e6bcb58aa: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790363087086945-787-0/minicluster-data/ts-2/wal/wals/41e0565c34a74925916fd7e5170b4602
I20260925 19:05:09.171497 5905 ts_tablet_manager.cc:1939] T 41e0565c34a74925916fd7e5170b4602 P 43622c34489e404999d0916e6bcb58aa: Deleting consensus metadata
I20260925 19:05:09.172039 5402 catalog_manager.cc:4841] TS 43622c34489e404999d0916e6bcb58aa (127.0.196.195:33599): tablet 41e0565c34a74925916fd7e5170b4602 (table test-table [id=a3d8f9edbaf84bbd94691c4b16fde0ce]) successfully deleted
W20260925 19:05:09.173535 5399 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.196.193:43527 (error 108)
I20260925 19:05:09.174064 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:09.173927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:43527 (local address 127.0.0.1:52242)
0925 19:05:09.173985 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:09.174026 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.193:43527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 19:05:09.174058 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 5606
W20260925 19:05:09.174698 5399 catalog_manager.cc:4584] TS 55662971924f430daff6aea93acfafa8 (127.0.196.193:43527): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 41e0565c34a74925916fd7e5170b4602: Network error: Client connection negotiation failed: client connection to 127.0.196.193:43527: connect: Connection refused (error 111)
I20260925 19:05:09.179452 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 5737
I20260925 19:05:09.184012 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 5384
2026-09-25T19:05:09Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (961 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 19:05:09.204270 787 mini_oidc.cc:152] Starting JWKS server
I20260925 19:05:09.205480 787 webserver.cc:460] Webserver started at https://127.0.0.1:34529/ using document root <none> and password file <none>
I20260925 19:05:09.205622 787 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 19:05:09.205837 787 webserver.cc:460] Webserver started at http://127.0.0.1:36419/ using document root <none> and password file <none>
2026-09-25T19:05:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:05:09Z Disabled control of system clock
I20260925 19:05:09.214278 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:45013
--webserver_interface=127.0.196.254
--webserver_port=0
--builtin_ntp_servers=127.0.196.212:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:45013
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34529/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363087086945-787-0/testchainca.cert with env {}
W20260925 19:05:09.297109 5912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:09.297350 5912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:09.297376 5912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:09.298574 5912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:05:09.298632 5912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:09.298646 5912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:09.298664 5912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:05:09.298676 5912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 19:05:09.298691 5912 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 19:05:09.298709 5912 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34529/jwks.json
I20260925 19:05:09.300309 5912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:45013
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:45013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34529/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363087086945-787-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:09.300534 5912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:09.300714 5912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:09.305686 5920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:09.305703 5918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:09.305757 5917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:09.305855 5912 server_base.cc:1034] running on GCE node
I20260925 19:05:09.306156 5912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:09.306486 5912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:09.307660 5912 hybrid_clock.cc:648] HybridClock initialized: now 1790363109307635 us; error 43 us; skew 500 ppm
I20260925 19:05:09.309036 5912 webserver.cc:460] Webserver started at http://127.0.196.254:40583/ using document root <none> and password file <none>
I20260925 19:05:09.309347 5912 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:09.309397 5912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:09.309567 5912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:09.310745 5912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "557c1879a3444dfcb1ea4f32d23bd63d"
format_stamp: "Formatted at 2026-09-25 19:05:09 on dist-test-slave-rn98"
server_key: "05eabf2acd616d79cbccf5389536175d"
server_key_iv: "881f364adf6895e8cb0084b443ebf0a4"
server_key_version: "encryptionkey@0"
I20260925 19:05:09.311131 5912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "557c1879a3444dfcb1ea4f32d23bd63d"
format_stamp: "Formatted at 2026-09-25 19:05:09 on dist-test-slave-rn98"
server_key: "05eabf2acd616d79cbccf5389536175d"
server_key_iv: "881f364adf6895e8cb0084b443ebf0a4"
server_key_version: "encryptionkey@0"
I20260925 19:05:09.312461 5912 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:05:09.313230 5926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:09.313412 5912 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:09.313505 5912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "557c1879a3444dfcb1ea4f32d23bd63d"
format_stamp: "Formatted at 2026-09-25 19:05:09 on dist-test-slave-rn98"
server_key: "05eabf2acd616d79cbccf5389536175d"
server_key_iv: "881f364adf6895e8cb0084b443ebf0a4"
server_key_version: "encryptionkey@0"
I20260925 19:05:09.313582 5912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:09.325246 5912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:09.325595 5912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:09.325757 5912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:09.329895 5912 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:45013
I20260925 19:05:09.329928 5978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:45013 every 8 connection(s)
I20260925 19:05:09.330296 5912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:05:09.331333 5979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:09.333595 5979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 557c1879a3444dfcb1ea4f32d23bd63d: Bootstrap starting.
I20260925 19:05:09.334180 5979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 557c1879a3444dfcb1ea4f32d23bd63d: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:09.334422 5979 log.cc:824] T 00000000000000000000000000000000 P 557c1879a3444dfcb1ea4f32d23bd63d: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:09.335127 5979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 557c1879a3444dfcb1ea4f32d23bd63d: No bootstrap required, opened a new log
I20260925 19:05:09.336758 5979 raft_consensus.cc:348] T 00000000000000000000000000000000 P 557c1879a3444dfcb1ea4f32d23bd63d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "557c1879a3444dfcb1ea4f32d23bd63d" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 45013 } }
I20260925 19:05:09.336936 5979 raft_consensus.cc:374] T 00000000000000000000000000000000 P 557c1879a3444dfcb1ea4f32d23bd63d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:09.336985 5979 raft_consensus.cc:729] T 00000000000000000000000000000000 P 557c1879a3444dfcb1ea4f32d23bd63d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 557c1879a3444dfcb1ea4f32d23bd63d, State: Initialized, Role: FOLLOWER
I20260925 19:05:09.337134 5979 consensus_queue.cc:260] T 00000000000000000000000000000000 P 557c1879a3444dfcb1ea4f32d23bd63d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "557c1879a3444dfcb1ea4f32d23bd63d" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 45013 } }
I20260925 19:05:09.337217 5979 raft_consensus.cc:388] T 00000000000000000000000000000000 P 557c1879a3444dfcb1ea4f32d23bd63d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:05:09.337260 5979 raft_consensus.cc:482] T 00000000000000000000000000000000 P 557c1879a3444dfcb1ea4f32d23bd63d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:05:09.337312 5979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 557c1879a3444dfcb1ea4f32d23bd63d [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:09.337838 5979 raft_consensus.cc:504] T 00000000000000000000000000000000 P 557c1879a3444dfcb1ea4f32d23bd63d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "557c1879a3444dfcb1ea4f32d23bd63d" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 45013 } }
I20260925 19:05:09.337958 5979 leader_election.cc:302] T 00000000000000000000000000000000 P 557c1879a3444dfcb1ea4f32d23bd63d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 557c1879a3444dfcb1ea4f32d23bd63d; no voters:
I20260925 19:05:09.338148 5979 leader_election.cc:288] T 00000000000000000000000000000000 P 557c1879a3444dfcb1ea4f32d23bd63d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:05:09.338212 5982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 557c1879a3444dfcb1ea4f32d23bd63d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:05:09.338408 5982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 557c1879a3444dfcb1ea4f32d23bd63d [term 1 LEADER]: Becoming Leader. State: Replica: 557c1879a3444dfcb1ea4f32d23bd63d, State: Running, Role: LEADER
I20260925 19:05:09.338418 5979 sys_catalog.cc:564] T 00000000000000000000000000000000 P 557c1879a3444dfcb1ea4f32d23bd63d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:05:09.338538 5982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 557c1879a3444dfcb1ea4f32d23bd63d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "557c1879a3444dfcb1ea4f32d23bd63d" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 45013 } }
I20260925 19:05:09.338661 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 5912
I20260925 19:05:09.338758 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:05:09.338954 787 external_mini_cluster.cc:1428] Setting key 2fc09500e74b4753e1e6df12bf1c3d77
I20260925 19:05:09.339893 5984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 557c1879a3444dfcb1ea4f32d23bd63d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 557c1879a3444dfcb1ea4f32d23bd63d. Latest consensus state: current_term: 1 leader_uuid: "557c1879a3444dfcb1ea4f32d23bd63d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "557c1879a3444dfcb1ea4f32d23bd63d" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 45013 } } }
I20260925 19:05:09.339974 5984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 557c1879a3444dfcb1ea4f32d23bd63d [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:09.340142 5983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 557c1879a3444dfcb1ea4f32d23bd63d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "557c1879a3444dfcb1ea4f32d23bd63d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "557c1879a3444dfcb1ea4f32d23bd63d" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 45013 } } }
I20260925 19:05:09.340197 5983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 557c1879a3444dfcb1ea4f32d23bd63d [sys.catalog]: This master's current role is: LEADER
W20260925 19:05:09.340440 5999 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 557c1879a3444dfcb1ea4f32d23bd63d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:05:09.340503 5999 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:05:09.340559 6000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:05:09.340740 6000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:05:09.342986 6000 catalog_manager.cc:1347] Generated new cluster ID: f390d757eb5e4f5491347d57ea081a51
I20260925 19:05:09.343050 6000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:05:09.345440 5995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:09.339608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40572 (local address 127.0.196.254:45013)
0925 19:05:09.340058 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:09.340067 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:09.340099 (+ 32us) server_negotiation.cc:407] Connection header received
0925 19:05:09.340248 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:09.340255 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:09.340328 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:09.340547 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:09.340860 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:09.342390 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:09.343395 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:09.343824 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:09.343988 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:09.344766 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:09.344770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:09.344770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:05:09.344948 (+ 178us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:09.344983 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:05:09.344994 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:09.345067 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:09.345140 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 19:05:09.345253 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":69,"threads_started":1}
I20260925 19:05:09.346338 787 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:05:09.357774 6000 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:05:09.358397 6000 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:05:09.361487 6000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 557c1879a3444dfcb1ea4f32d23bd63d: Generated new TSK 0
I20260925 19:05:09.361703 6000 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 19:05:09.391515 6009 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 19:05:09.391636 6009 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260925 19:05:09.392155 6008 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:09.369465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:45013 (local address 127.0.0.1:40584)
0925 19:05:09.369649 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:09.369663 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:09.369754 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:09.369925 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:09.369926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:09.369929 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0925 19:05:09.370007 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:09.370010 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:09.370675 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:09.370676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:09.371747 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:09.371750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:09.372100 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:09.372101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:09.372175 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:09.372199 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 19:05:09.391825 (+ 19626us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34529/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 19:05:09.391911 (+ 86us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.196.254:45013: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34529/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.196.254:45013: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34529/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 19:05:09.392473 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 5912
I20260925 19:05:09.392477 5995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:09.369559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40584 (local address 127.0.196.254:45013)
0925 19:05:09.369604 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:09.369608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:09.369738 (+ 130us) server_negotiation.cc:407] Connection header received
0925 19:05:09.369849 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:09.369849 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:09.369859 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:09.369918 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0925 19:05:09.370068 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:09.370552 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:09.371835 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:09.372017 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:09.372069 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:09.372305 (+ 236us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 19:05:09.391635 (+ 19330us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34529/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 19:05:09.392410 (+ 775us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40584: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34529/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 19:05:09.392526 5995 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40584: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34529/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
2026-09-25T19:05:09Z chronyd exiting
I20260925 19:05:09.428900 787 test_util.cc:175] -----------------------------------------------
I20260925 19:05:09.428983 787 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790363087086945-787-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (225 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 19:05:09.429741 787 mini_oidc.cc:152] Starting JWKS server
I20260925 19:05:09.430435 787 webserver.cc:460] Webserver started at https://127.0.0.1:38477/ using document root <none> and password file <none>
I20260925 19:05:09.430477 787 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 19:05:09.430625 787 webserver.cc:460] Webserver started at http://127.0.0.1:42255/ using document root <none> and password file <none>
2026-09-25T19:05:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:05:09Z Disabled control of system clock
I20260925 19:05:09.438318 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:37025
--webserver_interface=127.0.196.254
--webserver_port=0
--builtin_ntp_servers=127.0.196.212:34547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:37025
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38477/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363087086945-787-0/testchainca.cert with env {}
W20260925 19:05:09.522303 6015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:09.522558 6015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:09.522598 6015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:09.523913 6015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:05:09.524008 6015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:09.524029 6015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:09.524051 6015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:05:09.524077 6015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 19:05:09.524111 6015 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 19:05:09.524132 6015 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38477/jwks.json
I20260925 19:05:09.525761 6015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:34547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:37025
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:37025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38477/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363087086945-787-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:09.526078 6015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:09.526326 6015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:09.531407 6021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:09.531452 6023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:09.531590 6020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:09.531610 6015 server_base.cc:1034] running on GCE node
I20260925 19:05:09.531993 6015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:09.532348 6015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:09.533499 6015 hybrid_clock.cc:648] HybridClock initialized: now 1790363109533479 us; error 45 us; skew 500 ppm
I20260925 19:05:09.534868 6015 webserver.cc:460] Webserver started at http://127.0.196.254:39587/ using document root <none> and password file <none>
I20260925 19:05:09.535138 6015 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:09.535235 6015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:09.535390 6015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:09.536494 6015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "c8cdddc88650437db34319ee6cab5bcf"
format_stamp: "Formatted at 2026-09-25 19:05:09 on dist-test-slave-rn98"
server_key: "289c4adde917f89bead7825369485566"
server_key_iv: "debcc954f881abf2dbd03d6b85732ffc"
server_key_version: "encryptionkey@0"
I20260925 19:05:09.536867 6015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "c8cdddc88650437db34319ee6cab5bcf"
format_stamp: "Formatted at 2026-09-25 19:05:09 on dist-test-slave-rn98"
server_key: "289c4adde917f89bead7825369485566"
server_key_iv: "debcc954f881abf2dbd03d6b85732ffc"
server_key_version: "encryptionkey@0"
I20260925 19:05:09.538264 6015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:05:09.539104 6029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:09.539309 6015 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:05:09.539417 6015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "c8cdddc88650437db34319ee6cab5bcf"
format_stamp: "Formatted at 2026-09-25 19:05:09 on dist-test-slave-rn98"
server_key: "289c4adde917f89bead7825369485566"
server_key_iv: "debcc954f881abf2dbd03d6b85732ffc"
server_key_version: "encryptionkey@0"
I20260925 19:05:09.539494 6015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:09.557255 6015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:09.557657 6015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:09.557839 6015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:09.562093 6015 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:37025
I20260925 19:05:09.562132 6081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:37025 every 8 connection(s)
I20260925 19:05:09.562573 6015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:05:09.563601 6082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:09.565982 6082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8cdddc88650437db34319ee6cab5bcf: Bootstrap starting.
I20260925 19:05:09.566565 6082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8cdddc88650437db34319ee6cab5bcf: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:09.566908 6082 log.cc:824] T 00000000000000000000000000000000 P c8cdddc88650437db34319ee6cab5bcf: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:09.567718 6082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8cdddc88650437db34319ee6cab5bcf: No bootstrap required, opened a new log
I20260925 19:05:09.569383 6082 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8cdddc88650437db34319ee6cab5bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cdddc88650437db34319ee6cab5bcf" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37025 } }
I20260925 19:05:09.569528 6082 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8cdddc88650437db34319ee6cab5bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:09.569551 6082 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8cdddc88650437db34319ee6cab5bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8cdddc88650437db34319ee6cab5bcf, State: Initialized, Role: FOLLOWER
I20260925 19:05:09.569701 6082 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8cdddc88650437db34319ee6cab5bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cdddc88650437db34319ee6cab5bcf" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37025 } }
I20260925 19:05:09.569788 6082 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8cdddc88650437db34319ee6cab5bcf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:05:09.569811 6082 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8cdddc88650437db34319ee6cab5bcf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:05:09.569840 6082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8cdddc88650437db34319ee6cab5bcf [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:09.570361 6082 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8cdddc88650437db34319ee6cab5bcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cdddc88650437db34319ee6cab5bcf" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37025 } }
I20260925 19:05:09.570446 6082 leader_election.cc:302] T 00000000000000000000000000000000 P c8cdddc88650437db34319ee6cab5bcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8cdddc88650437db34319ee6cab5bcf; no voters:
I20260925 19:05:09.570580 6082 leader_election.cc:288] T 00000000000000000000000000000000 P c8cdddc88650437db34319ee6cab5bcf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:05:09.570679 6085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8cdddc88650437db34319ee6cab5bcf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:05:09.570871 6082 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8cdddc88650437db34319ee6cab5bcf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:05:09.570914 6085 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8cdddc88650437db34319ee6cab5bcf [term 1 LEADER]: Becoming Leader. State: Replica: c8cdddc88650437db34319ee6cab5bcf, State: Running, Role: LEADER
I20260925 19:05:09.571010 6085 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8cdddc88650437db34319ee6cab5bcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cdddc88650437db34319ee6cab5bcf" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37025 } }
I20260925 19:05:09.571457 6087 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8cdddc88650437db34319ee6cab5bcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8cdddc88650437db34319ee6cab5bcf. Latest consensus state: current_term: 1 leader_uuid: "c8cdddc88650437db34319ee6cab5bcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cdddc88650437db34319ee6cab5bcf" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37025 } } }
I20260925 19:05:09.571537 6087 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8cdddc88650437db34319ee6cab5bcf [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:09.571537 6086 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8cdddc88650437db34319ee6cab5bcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8cdddc88650437db34319ee6cab5bcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cdddc88650437db34319ee6cab5bcf" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37025 } } }
I20260925 19:05:09.571640 6086 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8cdddc88650437db34319ee6cab5bcf [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:09.571978 6095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:05:09.572521 6095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:05:09.572755 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 6015
I20260925 19:05:09.572856 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:05:09.573036 787 external_mini_cluster.cc:1428] Setting key 02b660f7c33dd2b1c0fda87943627f4c
I20260925 19:05:09.574314 6095 catalog_manager.cc:1347] Generated new cluster ID: df36a9f07f8a40d8b57e70cbc3e35708
I20260925 19:05:09.574373 6095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:05:09.579521 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:09.573474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52742 (local address 127.0.196.254:37025)
0925 19:05:09.573661 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:09.573672 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:09.573758 (+ 86us) server_negotiation.cc:407] Connection header received
0925 19:05:09.573931 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:09.573937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:09.573993 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:09.574131 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:09.574345 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:09.575800 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:09.576994 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:09.577625 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:09.577762 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:09.578537 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:09.578541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:09.578541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:05:09.578804 (+ 263us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:09.578848 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:05:09.578863 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:09.578927 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:09.579031 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 19:05:09.579113 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":84,"threads_started":1}
I20260925 19:05:09.580523 787 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:05:09.582195 6095 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:05:09.582784 6095 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:05:09.590297 6095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8cdddc88650437db34319ee6cab5bcf: Generated new TSK 0
I20260925 19:05:09.590521 6095 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 19:05:09.612206 6112 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 19:05:09.612329 6112 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 19:05:09.612627 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:09.597957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52758 (local address 127.0.196.254:37025)
0925 19:05:09.598009 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:09.598012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:09.598104 (+ 92us) server_negotiation.cc:407] Connection header received
0925 19:05:09.598170 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:09.598171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:09.598184 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:09.598272 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0925 19:05:09.598377 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:09.599182 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:09.599972 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:09.600255 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:09.600332 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:09.600567 (+ 235us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 19:05:09.612373 (+ 11806us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38477/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 19:05:09.612535 (+ 162us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52758: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38477/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":30}
W20260925 19:05:09.612695 6104 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52758: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38477/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 19:05:09.612676 6111 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:09.597877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:37025 (local address 127.0.0.1:52758)
0925 19:05:09.598027 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:09.598039 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:09.598109 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:09.598268 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:09.598269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:09.598270 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0925 19:05:09.598316 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:09.598318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:09.599289 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:09.599290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:09.599887 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:09.599889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:09.600368 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:09.600369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:09.600440 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:09.600447 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 19:05:09.612494 (+ 12047us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38477/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 19:05:09.612609 (+ 115us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.196.254:37025: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38477/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":101,"threads_started":1}
I20260925 19:05:09.613015 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 6015
2026-09-25T19:05:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (220 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 19:05:09.650346 787 mini_oidc.cc:152] Starting JWKS server
I20260925 19:05:09.651105 787 webserver.cc:460] Webserver started at https://127.0.0.1:45341/ using document root <none> and password file <none>
I20260925 19:05:09.651151 787 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 19:05:09.651300 787 webserver.cc:460] Webserver started at http://127.0.0.1:38597/ using document root <none> and password file <none>
2026-09-25T19:05:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:05:09Z Disabled control of system clock
I20260925 19:05:09.659439 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:38257
--webserver_interface=127.0.196.254
--webserver_port=0
--builtin_ntp_servers=127.0.196.212:39343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:38257
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45341/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 19:05:09.746542 6118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:09.746891 6118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:09.746930 6118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:09.748153 6118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:05:09.748225 6118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:09.748250 6118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:09.748281 6118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:05:09.748306 6118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 19:05:09.748332 6118 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 19:05:09.748358 6118 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45341/jwks.json
I20260925 19:05:09.749744 6118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:39343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:38257
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:38257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45341/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:09.750030 6118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:09.750361 6118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:05:09.755440 6118 server_base.cc:1034] running on GCE node
W20260925 19:05:09.755441 6123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:09.755481 6124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:09.755605 6126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:09.755954 6118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:09.756287 6118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:09.757467 6118 hybrid_clock.cc:648] HybridClock initialized: now 1790363109757457 us; error 43 us; skew 500 ppm
I20260925 19:05:09.758922 6118 webserver.cc:460] Webserver started at http://127.0.196.254:39939/ using document root <none> and password file <none>
I20260925 19:05:09.759337 6118 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:09.759403 6118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:09.759542 6118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:09.760816 6118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "e8cd0ad896ad40f4b2ee8fda29e8fe4e"
format_stamp: "Formatted at 2026-09-25 19:05:09 on dist-test-slave-rn98"
server_key: "832133cdfc9efecf4064ee91ded3721b"
server_key_iv: "19c2d396b54963e1b8c98d86990f5cd5"
server_key_version: "encryptionkey@0"
I20260925 19:05:09.761209 6118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "e8cd0ad896ad40f4b2ee8fda29e8fe4e"
format_stamp: "Formatted at 2026-09-25 19:05:09 on dist-test-slave-rn98"
server_key: "832133cdfc9efecf4064ee91ded3721b"
server_key_iv: "19c2d396b54963e1b8c98d86990f5cd5"
server_key_version: "encryptionkey@0"
I20260925 19:05:09.762591 6118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:05:09.763548 6132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:09.763758 6118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:09.763866 6118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "e8cd0ad896ad40f4b2ee8fda29e8fe4e"
format_stamp: "Formatted at 2026-09-25 19:05:09 on dist-test-slave-rn98"
server_key: "832133cdfc9efecf4064ee91ded3721b"
server_key_iv: "19c2d396b54963e1b8c98d86990f5cd5"
server_key_version: "encryptionkey@0"
I20260925 19:05:09.763965 6118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:09.777621 6118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:09.777992 6118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:09.778165 6118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:09.782413 6118 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:38257
I20260925 19:05:09.782465 6184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:38257 every 8 connection(s)
I20260925 19:05:09.782902 6118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:05:09.783592 6185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:09.784379 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 6118
I20260925 19:05:09.784487 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:05:09.784685 787 external_mini_cluster.cc:1428] Setting key a90b19e7d6b4d4e56a4ec4bbf4f95831
I20260925 19:05:09.787825 6185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8cd0ad896ad40f4b2ee8fda29e8fe4e: Bootstrap starting.
I20260925 19:05:09.788573 6185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8cd0ad896ad40f4b2ee8fda29e8fe4e: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:09.788909 6185 log.cc:824] T 00000000000000000000000000000000 P e8cd0ad896ad40f4b2ee8fda29e8fe4e: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:09.789747 6185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8cd0ad896ad40f4b2ee8fda29e8fe4e: No bootstrap required, opened a new log
I20260925 19:05:09.790982 6188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:09.785219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.0.196.254:38257)
0925 19:05:09.785430 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:09.785438 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:09.785464 (+ 26us) server_negotiation.cc:407] Connection header received
0925 19:05:09.785900 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:09.785906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:09.785972 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:09.786088 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:09.786361 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:09.787718 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:09.788957 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:09.789426 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:09.789520 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:09.790264 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:09.790268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:09.790268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:05:09.790432 (+ 164us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:09.790464 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:05:09.790474 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:09.790526 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:09.790695 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 19:05:09.790804 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260925 19:05:09.792043 787 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:05:09.792199 6185 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8cd0ad896ad40f4b2ee8fda29e8fe4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8cd0ad896ad40f4b2ee8fda29e8fe4e" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38257 } }
I20260925 19:05:09.792366 6185 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8cd0ad896ad40f4b2ee8fda29e8fe4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:09.792397 6185 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8cd0ad896ad40f4b2ee8fda29e8fe4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8cd0ad896ad40f4b2ee8fda29e8fe4e, State: Initialized, Role: FOLLOWER
I20260925 19:05:09.792531 6185 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8cd0ad896ad40f4b2ee8fda29e8fe4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8cd0ad896ad40f4b2ee8fda29e8fe4e" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38257 } }
I20260925 19:05:09.792610 6185 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8cd0ad896ad40f4b2ee8fda29e8fe4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:05:09.792639 6185 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8cd0ad896ad40f4b2ee8fda29e8fe4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:05:09.792680 6185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8cd0ad896ad40f4b2ee8fda29e8fe4e [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:09.793383 6185 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8cd0ad896ad40f4b2ee8fda29e8fe4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8cd0ad896ad40f4b2ee8fda29e8fe4e" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38257 } }
I20260925 19:05:09.793474 6185 leader_election.cc:302] T 00000000000000000000000000000000 P e8cd0ad896ad40f4b2ee8fda29e8fe4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8cd0ad896ad40f4b2ee8fda29e8fe4e; no voters:
I20260925 19:05:09.793614 6185 leader_election.cc:288] T 00000000000000000000000000000000 P e8cd0ad896ad40f4b2ee8fda29e8fe4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:05:09.793701 6190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8cd0ad896ad40f4b2ee8fda29e8fe4e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:05:09.793856 6190 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8cd0ad896ad40f4b2ee8fda29e8fe4e [term 1 LEADER]: Becoming Leader. State: Replica: e8cd0ad896ad40f4b2ee8fda29e8fe4e, State: Running, Role: LEADER
I20260925 19:05:09.793884 6185 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8cd0ad896ad40f4b2ee8fda29e8fe4e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:05:09.793942 6190 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8cd0ad896ad40f4b2ee8fda29e8fe4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8cd0ad896ad40f4b2ee8fda29e8fe4e" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38257 } }
I20260925 19:05:09.794283 6192 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8cd0ad896ad40f4b2ee8fda29e8fe4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8cd0ad896ad40f4b2ee8fda29e8fe4e. Latest consensus state: current_term: 1 leader_uuid: "e8cd0ad896ad40f4b2ee8fda29e8fe4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8cd0ad896ad40f4b2ee8fda29e8fe4e" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38257 } } }
I20260925 19:05:09.794320 6191 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8cd0ad896ad40f4b2ee8fda29e8fe4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8cd0ad896ad40f4b2ee8fda29e8fe4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8cd0ad896ad40f4b2ee8fda29e8fe4e" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38257 } } }
I20260925 19:05:09.794378 6192 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8cd0ad896ad40f4b2ee8fda29e8fe4e [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:09.794413 6191 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8cd0ad896ad40f4b2ee8fda29e8fe4e [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:09.794580 6197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:05:09.795112 6197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:05:09.796967 6197 catalog_manager.cc:1347] Generated new cluster ID: 187af6113eb04732bc4d20653adbd8e2
I20260925 19:05:09.797027 6197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:05:09.815822 6197 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:05:09.816440 6197 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:05:09.825120 6197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8cd0ad896ad40f4b2ee8fda29e8fe4e: Generated new TSK 0
I20260925 19:05:09.825371 6197 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:05:09.835726 6188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:09.831794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60674 (local address 127.0.196.254:38257)
0925 19:05:09.831862 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:09.831865 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:09.831966 (+ 101us) server_negotiation.cc:407] Connection header received
0925 19:05:09.832043 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:09.832044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:09.832058 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:09.832146 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0925 19:05:09.832259 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:09.833031 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:09.833888 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:09.834244 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:09.834316 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:09.834500 (+ 184us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 19:05:09.835507 (+ 1007us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45341/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 19:05:09.835645 (+ 138us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60674: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45341/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 19:05:09.835796 6188 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60674: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45341/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 19:05:09.835851 6214 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:09.831741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38257 (local address 127.0.0.1:60674)
0925 19:05:09.831882 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:09.831893 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:09.831962 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:09.832145 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:09.832146 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:09.832148 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 19:05:09.832197 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:09.832198 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:09.833142 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:09.833143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:09.833801 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:09.833803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:09.834336 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:09.834337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:09.834388 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:09.834393 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 19:05:09.835621 (+ 1228us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45341/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 19:05:09.835730 (+ 109us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.196.254:38257: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45341/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260925 19:05:09.836202 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 6118
E20260925 19:05:09.850152 6215 webserver.cc:533] Webserver: sslize failed:
E20260925 19:05:09.850242 6215 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T19:05:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (224 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T19:05:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:05:09Z Disabled control of system clock
I20260925 19:05:09.882568 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:43219
--webserver_interface=127.0.196.254
--webserver_port=0
--builtin_ntp_servers=127.0.196.212:32851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:43219
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363087086945-787-0/testchainca.cert with env {}
W20260925 19:05:09.964020 6219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:09.964249 6219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:09.964273 6219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:09.965397 6219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:05:09.965433 6219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:09.965446 6219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:09.965459 6219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:05:09.965471 6219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 19:05:09.965483 6219 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 19:05:09.965494 6219 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 19:05:09.967056 6219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:32851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:43219
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:43219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363087086945-787-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:09.967272 6219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:09.967437 6219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:09.972608 6225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:09.972561 6224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:09.972631 6227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:09.973065 6219 server_base.cc:1034] running on GCE node
I20260925 19:05:09.973316 6219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:09.973641 6219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:09.974856 6219 hybrid_clock.cc:648] HybridClock initialized: now 1790363109974843 us; error 31 us; skew 500 ppm
I20260925 19:05:09.976408 6219 webserver.cc:460] Webserver started at http://127.0.196.254:38403/ using document root <none> and password file <none>
I20260925 19:05:09.976688 6219 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:09.976742 6219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:09.976886 6219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:09.978119 6219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "4fb40df036d84f679c776191581aee54"
format_stamp: "Formatted at 2026-09-25 19:05:09 on dist-test-slave-rn98"
server_key: "8fab0bd174d07c3538ae0c0a5be6bba2"
server_key_iv: "3cceb6d710f706da9df9c5610eaecde8"
server_key_version: "encryptionkey@0"
I20260925 19:05:09.978545 6219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "4fb40df036d84f679c776191581aee54"
format_stamp: "Formatted at 2026-09-25 19:05:09 on dist-test-slave-rn98"
server_key: "8fab0bd174d07c3538ae0c0a5be6bba2"
server_key_iv: "3cceb6d710f706da9df9c5610eaecde8"
server_key_version: "encryptionkey@0"
I20260925 19:05:09.980146 6219 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:05:09.980981 6233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:09.981177 6219 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:05:09.981292 6219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "4fb40df036d84f679c776191581aee54"
format_stamp: "Formatted at 2026-09-25 19:05:09 on dist-test-slave-rn98"
server_key: "8fab0bd174d07c3538ae0c0a5be6bba2"
server_key_iv: "3cceb6d710f706da9df9c5610eaecde8"
server_key_version: "encryptionkey@0"
I20260925 19:05:09.981375 6219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:09.999159 6219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:09.999536 6219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:09.999714 6219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:10.004508 6219 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:43219
I20260925 19:05:10.004561 6285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:43219 every 8 connection(s)
I20260925 19:05:10.004976 6219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:05:10.006001 6286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:10.007194 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 6219
I20260925 19:05:10.007323 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:05:10.007519 787 external_mini_cluster.cc:1428] Setting key a58121fb5efa561f1284262071cc9188
I20260925 19:05:10.008759 6286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fb40df036d84f679c776191581aee54: Bootstrap starting.
I20260925 19:05:10.009483 6286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fb40df036d84f679c776191581aee54: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:10.009775 6286 log.cc:824] T 00000000000000000000000000000000 P 4fb40df036d84f679c776191581aee54: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:10.010528 6286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fb40df036d84f679c776191581aee54: No bootstrap required, opened a new log
I20260925 19:05:10.012864 6286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fb40df036d84f679c776191581aee54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb40df036d84f679c776191581aee54" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 43219 } }
I20260925 19:05:10.013051 6286 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fb40df036d84f679c776191581aee54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:10.013092 6286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fb40df036d84f679c776191581aee54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fb40df036d84f679c776191581aee54, State: Initialized, Role: FOLLOWER
I20260925 19:05:10.013228 6286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fb40df036d84f679c776191581aee54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb40df036d84f679c776191581aee54" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 43219 } }
I20260925 19:05:10.013314 6286 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fb40df036d84f679c776191581aee54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:05:10.013337 6286 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fb40df036d84f679c776191581aee54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:05:10.013435 6286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fb40df036d84f679c776191581aee54 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:10.013610 6289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:10.008099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37386 (local address 127.0.196.254:43219)
0925 19:05:10.008308 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:10.008315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:10.008342 (+ 27us) server_negotiation.cc:407] Connection header received
0925 19:05:10.008766 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:10.008773 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:10.008839 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:10.009010 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:10.009251 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:10.010335 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:10.011454 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:10.011878 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:10.011978 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:10.012699 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:10.012702 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:10.012703 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:05:10.012877 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:10.012910 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:05:10.012920 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:10.012971 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:10.013100 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 19:05:10.013296 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
I20260925 19:05:10.014122 6286 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fb40df036d84f679c776191581aee54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb40df036d84f679c776191581aee54" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 43219 } }
I20260925 19:05:10.014233 6286 leader_election.cc:302] T 00000000000000000000000000000000 P 4fb40df036d84f679c776191581aee54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fb40df036d84f679c776191581aee54; no voters:
I20260925 19:05:10.014395 6286 leader_election.cc:288] T 00000000000000000000000000000000 P 4fb40df036d84f679c776191581aee54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:05:10.014415 6291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fb40df036d84f679c776191581aee54 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:05:10.014528 6291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fb40df036d84f679c776191581aee54 [term 1 LEADER]: Becoming Leader. State: Replica: 4fb40df036d84f679c776191581aee54, State: Running, Role: LEADER
I20260925 19:05:10.014643 6291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fb40df036d84f679c776191581aee54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb40df036d84f679c776191581aee54" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 43219 } }
I20260925 19:05:10.014638 787 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:05:10.014675 6286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fb40df036d84f679c776191581aee54 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:05:10.014978 6292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb40df036d84f679c776191581aee54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fb40df036d84f679c776191581aee54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb40df036d84f679c776191581aee54" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 43219 } } }
I20260925 19:05:10.015062 6292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb40df036d84f679c776191581aee54 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:10.015259 6293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb40df036d84f679c776191581aee54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fb40df036d84f679c776191581aee54. Latest consensus state: current_term: 1 leader_uuid: "4fb40df036d84f679c776191581aee54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb40df036d84f679c776191581aee54" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 43219 } } }
I20260925 19:05:10.015323 6293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb40df036d84f679c776191581aee54 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:10.015391 6296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:05:10.015933 6296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:05:10.017746 6296 catalog_manager.cc:1347] Generated new cluster ID: c1ad9976361f49c4b7c36c6aa2ab2925
I20260925 19:05:10.017838 6296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:05:10.048592 6296 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:05:10.049223 6296 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:05:10.055410 6296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fb40df036d84f679c776191581aee54: Generated new TSK 0
I20260925 19:05:10.055652 6296 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:05:10.094728 6289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:10.060935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37402 (local address 127.0.196.254:43219)
0925 19:05:10.060999 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:10.061002 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:10.061128 (+ 126us) server_negotiation.cc:407] Connection header received
0925 19:05:10.061170 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:10.061171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:10.061182 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:10.061253 (+ 71us) server_negotiation.cc:226] Negotiated authn=JWT
0925 19:05:10.061429 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:10.061960 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:10.063175 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:10.063386 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:10.063440 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:10.063645 (+ 205us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 19:05:10.094407 (+ 30762us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 19:05:10.094581 (+ 174us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37402: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 19:05:10.094820 6289 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37402: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 19:05:10.094858 6315 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:10.060862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:43219 (local address 127.0.0.1:37402)
0925 19:05:10.061028 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:10.061040 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:10.061105 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:10.061280 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:10.061281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:10.061283 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 19:05:10.061353 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:10.061356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:10.062070 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:10.062071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:10.063076 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:10.063080 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:10.063471 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:10.063472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:10.063535 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:10.063541 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 19:05:10.094575 (+ 31034us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 19:05:10.094753 (+ 178us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.196.254:43219: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":94,"threads_started":1}
I20260925 19:05:10.095291 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 6219
2026-09-25T19:05:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (238 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (89 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (90 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:05:10 dist-test-slave-rn98 krb5kdc[6327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:05:10 dist-test-slave-rn98 krb5kdc[6327](info): set up 2 sockets
Sep 25 19:05:10 dist-test-slave-rn98 krb5kdc[6327](info): commencing operation
krb5kdc: starting...
W20260925 19:05:11.339301 787 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:05:11 dist-test-slave-rn98 krb5kdc[6327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:05:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:05:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:05:11.464206 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:37313
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:32893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:37313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:11.547544 6340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:11.547765 6340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:11.547789 6340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:11.548982 6340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:05:11.549033 6340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:11.549052 6340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:11.549067 6340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:05:11.549079 6340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:05:11.550743 6340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:32893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:37313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:37313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:11.551012 6340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:11.551241 6340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:11.556234 6348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:11.556376 6340 server_base.cc:1034] running on GCE node
W20260925 19:05:11.556244 6345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:11.556252 6346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:11.556737 6340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:11.557047 6340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:11.558221 6340 hybrid_clock.cc:648] HybridClock initialized: now 1790363111558218 us; error 40 us; skew 500 ppm
Sep 25 19:05:11 dist-test-slave-rn98 krb5kdc[6327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:11.560838 6340 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:05:11.561331 6340 webserver.cc:460] Webserver started at http://127.0.196.254:41613/ using document root <none> and password file <none>
I20260925 19:05:11.561585 6340 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:11.561656 6340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:11.561806 6340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:11.562973 6340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "c9bada4ecb204e9b91ae363cee47245a"
format_stamp: "Formatted at 2026-09-25 19:05:11 on dist-test-slave-rn98"
server_key: "6dd1ae5a45eaaeec82d0e776159f92b7"
server_key_iv: "ea6854fdab9846b09ece894555497830"
server_key_version: "encryptionkey@0"
I20260925 19:05:11.563339 6340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "c9bada4ecb204e9b91ae363cee47245a"
format_stamp: "Formatted at 2026-09-25 19:05:11 on dist-test-slave-rn98"
server_key: "6dd1ae5a45eaaeec82d0e776159f92b7"
server_key_iv: "ea6854fdab9846b09ece894555497830"
server_key_version: "encryptionkey@0"
I20260925 19:05:11.564698 6340 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:11.565444 6355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:11.565630 6340 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:05:11.565737 6340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "c9bada4ecb204e9b91ae363cee47245a"
format_stamp: "Formatted at 2026-09-25 19:05:11 on dist-test-slave-rn98"
server_key: "6dd1ae5a45eaaeec82d0e776159f92b7"
server_key_iv: "ea6854fdab9846b09ece894555497830"
server_key_version: "encryptionkey@0"
I20260925 19:05:11.565814 6340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:11.577438 6340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:11.580144 6340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:11.580324 6340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:11.584843 6340 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:37313
I20260925 19:05:11.584885 6407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:37313 every 8 connection(s)
I20260925 19:05:11.585443 6340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:05:11.586179 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:11.588619 6408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9bada4ecb204e9b91ae363cee47245a: Bootstrap starting.
I20260925 19:05:11.589244 6408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9bada4ecb204e9b91ae363cee47245a: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:11.589445 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 6340
I20260925 19:05:11.589542 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:05:11.589622 6408 log.cc:824] T 00000000000000000000000000000000 P c9bada4ecb204e9b91ae363cee47245a: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:11.589737 787 external_mini_cluster.cc:1428] Setting key 47fb84706fc084c6a8facd5c3fb5b89d
I20260925 19:05:11.590539 6408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9bada4ecb204e9b91ae363cee47245a: No bootstrap required, opened a new log
I20260925 19:05:11.593287 6408 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9bada4ecb204e9b91ae363cee47245a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9bada4ecb204e9b91ae363cee47245a" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37313 } }
I20260925 19:05:11.593472 6408 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9bada4ecb204e9b91ae363cee47245a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:11.593508 6408 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9bada4ecb204e9b91ae363cee47245a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9bada4ecb204e9b91ae363cee47245a, State: Initialized, Role: FOLLOWER
I20260925 19:05:11.593658 6408 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9bada4ecb204e9b91ae363cee47245a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9bada4ecb204e9b91ae363cee47245a" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37313 } }
I20260925 19:05:11.593778 6408 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9bada4ecb204e9b91ae363cee47245a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:05:11.593817 6408 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9bada4ecb204e9b91ae363cee47245a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:05:11.593871 6408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9bada4ecb204e9b91ae363cee47245a [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:11.594517 6408 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9bada4ecb204e9b91ae363cee47245a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9bada4ecb204e9b91ae363cee47245a" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37313 } }
I20260925 19:05:11.594647 6408 leader_election.cc:302] T 00000000000000000000000000000000 P c9bada4ecb204e9b91ae363cee47245a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9bada4ecb204e9b91ae363cee47245a; no voters:
I20260925 19:05:11.594835 6408 leader_election.cc:288] T 00000000000000000000000000000000 P c9bada4ecb204e9b91ae363cee47245a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:05:11.594906 6413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9bada4ecb204e9b91ae363cee47245a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:05:11.595089 6408 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9bada4ecb204e9b91ae363cee47245a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:05:11.595089 6413 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9bada4ecb204e9b91ae363cee47245a [term 1 LEADER]: Becoming Leader. State: Replica: c9bada4ecb204e9b91ae363cee47245a, State: Running, Role: LEADER
I20260925 19:05:11.595254 6413 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9bada4ecb204e9b91ae363cee47245a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9bada4ecb204e9b91ae363cee47245a" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37313 } }
I20260925 19:05:11.595742 6414 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9bada4ecb204e9b91ae363cee47245a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9bada4ecb204e9b91ae363cee47245a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9bada4ecb204e9b91ae363cee47245a" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37313 } } }
I20260925 19:05:11.595989 6414 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9bada4ecb204e9b91ae363cee47245a [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:11.595901 6415 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9bada4ecb204e9b91ae363cee47245a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9bada4ecb204e9b91ae363cee47245a. Latest consensus state: current_term: 1 leader_uuid: "c9bada4ecb204e9b91ae363cee47245a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9bada4ecb204e9b91ae363cee47245a" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37313 } } }
I20260925 19:05:11.596261 6415 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9bada4ecb204e9b91ae363cee47245a [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:11.596628 6422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:05:11.597333 6422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 19:05:11 dist-test-slave-rn98 krb5kdc[6327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:05:11.599493 6422 catalog_manager.cc:1347] Generated new cluster ID: 111a0b1f77a741718ab2733fc856574a
I20260925 19:05:11.599591 6422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:05:11.601608 6412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:11.590275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39624 (local address 127.0.196.254:37313)
0925 19:05:11.590469 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:11.590478 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:11.590512 (+ 34us) server_negotiation.cc:407] Connection header received
0925 19:05:11.591099 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:11.591109 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:11.591201 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:11.591362 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:11.591958 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:11.593494 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:11.594393 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:11.595079 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:11.595551 (+ 472us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:11.598254 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:11.598259 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:11.598259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:11.598272 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:11.600011 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:11.600397 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:11.600400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:11.600400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:11.600497 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:11.600727 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:11.600728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:11.600728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:11.600994 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:11.601157 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:11.601278 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 19:05:11.601423 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":83,"threads_started":1}
I20260925 19:05:11.615671 6422 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:05:11.616217 6422 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:05:11.621426 6422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9bada4ecb204e9b91ae363cee47245a: Generated new TSK 0
I20260925 19:05:11.621657 6422 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
I20260925 19:05:11.657379 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:0
--local_ip_for_outbound_sockets=127.0.196.193
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:37313
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:32893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:11.739182 6434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:11.739394 6434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:11.739420 6434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:11.740550 6434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:11.740590 6434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:11.740617 6434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:05:11.742164 6434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:32893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:37313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:11.742365 6434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:11.742542 6434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:11.747704 6440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:11.747677 6442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:11.747725 6434 server_base.cc:1034] running on GCE node
W20260925 19:05:11.747694 6439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:11.748100 6434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:11.748315 6434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:11.749450 6434 hybrid_clock.cc:648] HybridClock initialized: now 1790363111749438 us; error 29 us; skew 500 ppm
Sep 25 19:05:11 dist-test-slave-rn98 krb5kdc[6327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:11.751996 6434 init.cc:377] Logged in from keytab as kudu/127.0.196.193@KRBTEST.COM (short username kudu)
I20260925 19:05:11.752394 6434 webserver.cc:460] Webserver started at http://127.0.196.193:36709/ using document root <none> and password file <none>
I20260925 19:05:11.752630 6434 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:11.752699 6434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:11.752856 6434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:11.753923 6434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-0/data/instance:
uuid: "24a937eb4316461182df0f1d8a031fe3"
format_stamp: "Formatted at 2026-09-25 19:05:11 on dist-test-slave-rn98"
server_key: "17233006241efdd377488c19032bd2b4"
server_key_iv: "6bdc64b20ec0f92fe38f47e3de2ce9da"
server_key_version: "encryptionkey@0"
I20260925 19:05:11.754292 6434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-0/wal/instance:
uuid: "24a937eb4316461182df0f1d8a031fe3"
format_stamp: "Formatted at 2026-09-25 19:05:11 on dist-test-slave-rn98"
server_key: "17233006241efdd377488c19032bd2b4"
server_key_iv: "6bdc64b20ec0f92fe38f47e3de2ce9da"
server_key_version: "encryptionkey@0"
I20260925 19:05:11.755761 6434 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:05:11.756534 6449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:11.756727 6434 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:11.756834 6434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "24a937eb4316461182df0f1d8a031fe3"
format_stamp: "Formatted at 2026-09-25 19:05:11 on dist-test-slave-rn98"
server_key: "17233006241efdd377488c19032bd2b4"
server_key_iv: "6bdc64b20ec0f92fe38f47e3de2ce9da"
server_key_version: "encryptionkey@0"
I20260925 19:05:11.756909 6434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:11.771569 6434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:11.774143 6434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:11.774354 6434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:11.774601 6434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:11.774964 6434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:11.775018 6434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:11.775053 6434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:11.775080 6434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:11.780682 6434 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:38019
I20260925 19:05:11.780743 6562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:38019 every 8 connection(s)
I20260925 19:05:11.781082 6434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
I20260925 19:05:11.782284 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 6434
I20260925 19:05:11.782377 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-0/wal/instance
I20260925 19:05:11.782596 787 external_mini_cluster.cc:1428] Setting key 3d091a2c0e34d7f95d62a6332901f89e
Sep 25 19:05:11 dist-test-slave-rn98 krb5kdc[6327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.196.194@KRBTEST.COM; defaulting to no policy
I20260925 19:05:11.792428 6412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:11.781856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:58079 (local address 127.0.196.254:37313)
0925 19:05:11.781946 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:11.781949 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:11.782161 (+ 212us) server_negotiation.cc:407] Connection header received
0925 19:05:11.783504 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:11.783506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:11.783520 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:11.783611 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:11.784226 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:11.785080 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:11.786215 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:11.786450 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:11.786554 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:11.789143 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:11.789145 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:11.789145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:11.789153 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:11.790992 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:11.791426 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:11.791427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:11.791428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:11.791479 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:11.791734 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:11.791735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:11.791735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:11.791857 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:11.791932 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:11.792242 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 19:05:11.792334 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 19:05:11.792639 6565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:11.781822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:37313 (local address 127.0.196.193:58079)
0925 19:05:11.781988 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:11.782029 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:11.783359 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:11.783672 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:11.783675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:11.783932 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:11.784129 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:11.784133 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:11.785191 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:11.785193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:11.786095 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:11.786097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:11.786566 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:11.786567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:11.786738 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:11.787345 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:11.787355 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:11.789023 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:11.791095 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:11.791102 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:11.791104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:11.791354 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:11.791539 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:11.791540 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:11.791540 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:11.791638 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:11.791934 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:11.791941 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:11.792078 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 19:05:11.792220 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 19:05:11.792385 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260925 19:05:11.793159 6563 heartbeater.cc:347] Connected to a master server at 127.0.196.254:37313
I20260925 19:05:11.793289 6563 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:11.793499 6563 heartbeater.cc:511] Master 127.0.196.254:37313 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.194@KRBTEST.COM" created.
I20260925 19:05:11.794075 6372 ts_manager.cc:194] Registered new tserver with Master: 24a937eb4316461182df0f1d8a031fe3 (127.0.196.193:38019)
I20260925 19:05:11.794770 6372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.193@KRBTEST.COM'} at 127.0.196.193:58079
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
I20260925 19:05:11.810009 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:0
--local_ip_for_outbound_sockets=127.0.196.194
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:37313
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:32893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:11.891718 6568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:11.891940 6568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:11.891964 6568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:11.893079 6568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:11.893114 6568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:11.893141 6568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:05:11.894703 6568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:32893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:37313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:11.894894 6568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:11.895131 6568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:11.900158 6573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:11.900235 6574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:11.900347 6568 server_base.cc:1034] running on GCE node
W20260925 19:05:11.900382 6576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:11.900574 6568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:11.900828 6568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:11.901973 6568 hybrid_clock.cc:648] HybridClock initialized: now 1790363111901959 us; error 24 us; skew 500 ppm
Sep 25 19:05:11 dist-test-slave-rn98 krb5kdc[6327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:11.904524 6568 init.cc:377] Logged in from keytab as kudu/127.0.196.194@KRBTEST.COM (short username kudu)
I20260925 19:05:11.904969 6568 webserver.cc:460] Webserver started at http://127.0.196.194:37915/ using document root <none> and password file <none>
I20260925 19:05:11.905200 6568 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:11.905246 6568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:11.905402 6568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:11.906491 6568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-1/data/instance:
uuid: "145cc25f31f6496e98cc7806039aa83c"
format_stamp: "Formatted at 2026-09-25 19:05:11 on dist-test-slave-rn98"
server_key: "11126e7d46da0e00818db7c3ef43aeb2"
server_key_iv: "91fa1b7e8a360187432cdd25bf870fd1"
server_key_version: "encryptionkey@0"
I20260925 19:05:11.906924 6568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-1/wal/instance:
uuid: "145cc25f31f6496e98cc7806039aa83c"
format_stamp: "Formatted at 2026-09-25 19:05:11 on dist-test-slave-rn98"
server_key: "11126e7d46da0e00818db7c3ef43aeb2"
server_key_iv: "91fa1b7e8a360187432cdd25bf870fd1"
server_key_version: "encryptionkey@0"
I20260925 19:05:11.908272 6568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:11.908973 6583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:11.909135 6568 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:05:11.909247 6568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "145cc25f31f6496e98cc7806039aa83c"
format_stamp: "Formatted at 2026-09-25 19:05:11 on dist-test-slave-rn98"
server_key: "11126e7d46da0e00818db7c3ef43aeb2"
server_key_iv: "91fa1b7e8a360187432cdd25bf870fd1"
server_key_version: "encryptionkey@0"
I20260925 19:05:11.909322 6568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:11.921519 6568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:11.923946 6568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:11.924117 6568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:11.924363 6568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:11.924681 6568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:11.924737 6568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:11.924780 6568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:11.924811 6568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:11.930859 6568 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:44055
I20260925 19:05:11.930905 6696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:44055 every 8 connection(s)
I20260925 19:05:11.931238 6568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
I20260925 19:05:11.935173 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 6568
I20260925 19:05:11.935256 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-1/wal/instance
I20260925 19:05:11.935446 787 external_mini_cluster.cc:1428] Setting key 3b3844576cf0242aaba79de9c5698498
Sep 25 19:05:11 dist-test-slave-rn98 krb5kdc[6327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:05:11.941493 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:11.931940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:37313 (local address 127.0.196.194:39699)
0925 19:05:11.932127 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:11.932171 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:11.933166 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:11.933412 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:11.933415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:11.933674 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:11.933864 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:11.933868 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:11.934510 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:11.934511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:11.935530 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:11.935533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:11.935979 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:11.935981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:11.936120 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:11.936653 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:11.936672 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:11.938239 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:11.940128 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:11.940136 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:11.940138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:11.940375 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:11.940595 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:11.940596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:11.940596 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:11.940697 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:11.940995 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:11.940999 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:11.941118 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 19:05:11.941200 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 19:05:11.941324 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":108,"threads_started":1}
I20260925 19:05:11.941484 6412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:11.931966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:39699 (local address 127.0.196.254:37313)
0925 19:05:11.932038 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:11.932040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:11.932271 (+ 231us) server_negotiation.cc:407] Connection header received
0925 19:05:11.933262 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:11.933263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:11.933272 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:11.933335 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:11.933953 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:11.934433 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:11.935603 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:11.935868 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:11.935970 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:11.938354 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:11.938356 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:11.938357 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:11.938366 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:11.940031 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:11.940455 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:11.940457 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:11.940457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:11.940523 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:11.940763 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:11.940765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:11.940765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:11.940922 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:11.941005 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:11.941253 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 19:05:11.941346 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 19:05:11.942086 6697 heartbeater.cc:347] Connected to a master server at 127.0.196.254:37313
I20260925 19:05:11.942173 6697 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:11.942354 6697 heartbeater.cc:511] Master 127.0.196.254:37313 requested a full tablet report, sending...
I20260925 19:05:11.942801 6372 ts_manager.cc:194] Registered new tserver with Master: 145cc25f31f6496e98cc7806039aa83c (127.0.196.194:44055)
I20260925 19:05:11.943224 6372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.194@KRBTEST.COM'} at 127.0.196.194:39699
WARNING: no policy specified for kudu/127.0.196.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
I20260925 19:05:11.963919 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:0
--local_ip_for_outbound_sockets=127.0.196.195
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:37313
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:32893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:12.045517 6702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:12.045790 6702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:12.045831 6702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:12.047073 6702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:12.047154 6702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:12.047211 6702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:05:12.048800 6702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:32893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:37313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:12.049088 6702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:12.049347 6702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:12.054658 6708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:12.054677 6707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:12.054706 6710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:12.055039 6702 server_base.cc:1034] running on GCE node
I20260925 19:05:12.055338 6702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:12.055604 6702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:12.056779 6702 hybrid_clock.cc:648] HybridClock initialized: now 1790363112056744 us; error 63 us; skew 500 ppm
Sep 25 19:05:12 dist-test-slave-rn98 krb5kdc[6327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:12.059691 6702 init.cc:377] Logged in from keytab as kudu/127.0.196.195@KRBTEST.COM (short username kudu)
I20260925 19:05:12.060307 6702 webserver.cc:460] Webserver started at http://127.0.196.195:36395/ using document root <none> and password file <none>
I20260925 19:05:12.060562 6702 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:12.060632 6702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:12.060789 6702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:12.061874 6702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-2/data/instance:
uuid: "881b55a1b3a14062a11c455209650bfb"
format_stamp: "Formatted at 2026-09-25 19:05:12 on dist-test-slave-rn98"
server_key: "aee9745b6755653efba949428d448a7a"
server_key_iv: "f28eda9b800aeb054ad44074c3de2e9c"
server_key_version: "encryptionkey@0"
I20260925 19:05:12.062251 6702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-2/wal/instance:
uuid: "881b55a1b3a14062a11c455209650bfb"
format_stamp: "Formatted at 2026-09-25 19:05:12 on dist-test-slave-rn98"
server_key: "aee9745b6755653efba949428d448a7a"
server_key_iv: "f28eda9b800aeb054ad44074c3de2e9c"
server_key_version: "encryptionkey@0"
I20260925 19:05:12.063755 6702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:05:12.064571 6717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:12.064782 6702 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:05:12.064898 6702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "881b55a1b3a14062a11c455209650bfb"
format_stamp: "Formatted at 2026-09-25 19:05:12 on dist-test-slave-rn98"
server_key: "aee9745b6755653efba949428d448a7a"
server_key_iv: "f28eda9b800aeb054ad44074c3de2e9c"
server_key_version: "encryptionkey@0"
I20260925 19:05:12.064980 6702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:12.072448 6702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:12.074548 6702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:12.074749 6702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:12.074992 6702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:12.075306 6702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:12.075362 6702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:12.075407 6702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:12.075433 6702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:12.081444 6702 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:45541
I20260925 19:05:12.081480 6830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:45541 every 8 connection(s)
I20260925 19:05:12.081873 6702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
Sep 25 19:05:12 dist-test-slave-rn98 krb5kdc[6327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:05:12.088784 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 6702
I20260925 19:05:12.088896 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/minicluster-data/ts-2/wal/instance
I20260925 19:05:12.089150 787 external_mini_cluster.cc:1428] Setting key 84c35e714d7f4f14d1836368a76ea050
I20260925 19:05:12.092904 6412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:12.082663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:52935 (local address 127.0.196.254:37313)
0925 19:05:12.082715 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:12.082718 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:12.082975 (+ 257us) server_negotiation.cc:407] Connection header received
0925 19:05:12.083986 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:12.083987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:12.084000 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:12.084058 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:12.084726 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:12.085242 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:12.086274 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:12.086484 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:12.086572 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:12.089070 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:12.089072 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:12.089072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:12.089080 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:12.091013 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:12.091512 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:12.091514 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:12.091515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:12.091584 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:12.091861 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:12.091862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:12.091863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:12.092008 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:12.092399 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:12.092724 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 19:05:12.092811 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:05:12.093122 6833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:12.082670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:37313 (local address 127.0.196.195:52935)
0925 19:05:12.082842 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:12.082881 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:12.083864 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:12.084140 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:12.084144 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:12.084414 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:12.084616 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:12.084620 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:12.085343 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:12.085344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:12.086172 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:12.086174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:12.086582 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:12.086583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:12.086772 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:12.087275 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:12.087284 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:12.088934 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:12.091111 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:12.091119 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:12.091122 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:12.091390 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:12.091656 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:12.091657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:12.091657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:12.091786 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:12.092382 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:12.092389 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:12.092557 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 19:05:12.092679 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 19:05:12.092856 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260925 19:05:12.093761 6831 heartbeater.cc:347] Connected to a master server at 127.0.196.254:37313
I20260925 19:05:12.093868 6831 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:12.094028 6831 heartbeater.cc:511] Master 127.0.196.254:37313 requested a full tablet report, sending...
I20260925 19:05:12.094508 6372 ts_manager.cc:194] Registered new tserver with Master: 881b55a1b3a14062a11c455209650bfb (127.0.196.195:45541)
I20260925 19:05:12.094930 6372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.195@KRBTEST.COM'} at 127.0.196.195:52935
I20260925 19:05:12.101188 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/test-admin.keytab.
Sep 25 19:05:12 dist-test-slave-rn98 krb5kdc[6327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:12.122520 787 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 19:05:12.124351 6838 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:12.123483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58158 (local address 127.0.196.193:38019)
0925 19:05:12.123679 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:12.123686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:12.123699 (+ 13us) server_negotiation.cc:407] Connection header received
0925 19:05:12.123732 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:12.123734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:12.123751 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:12.123851 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:12.124261 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58158: BlockingRecv error: recv got EOF from 127.0.0.1:58158 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":91,"threads_started":1}
W20260925 19:05:12.124464 787 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/krb5cc' of type FILE
Sep 25 19:05:12 dist-test-slave-rn98 krb5kdc[6327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:12.125571 787 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 19:05:12 dist-test-slave-rn98 krb5kdc[6327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:12.126550 787 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 19:05:12 dist-test-slave-rn98 krb5kdc[6327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.193@KRBTEST.COM
I20260925 19:05:12.134967 6838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:12.127220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58164 (local address 127.0.196.193:38019)
0925 19:05:12.127276 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:12.127278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:12.127288 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:05:12.127361 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:12.127361 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:12.127373 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:12.127424 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:12.127836 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:12.128602 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:12.129425 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:12.129770 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:12.129852 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:12.132069 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:12.132079 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:12.132079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:12.132087 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:12.133733 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:12.134114 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:12.134117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:12.134117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:12.134175 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:12.134372 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:12.134373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:12.134373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:12.134518 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:12.134701 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:12.134771 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 19:05:12.134850 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260925 19:05:12.136845 6838 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:12.136150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58168 (local address 127.0.196.193:38019)
0925 19:05:12.136204 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:12.136205 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:12.136288 (+ 83us) server_negotiation.cc:407] Connection header received
0925 19:05:12.136337 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:12.136338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:12.136346 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:12.136392 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:12.136795 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58168: BlockingRecv error: recv got EOF from 127.0.0.1:58168 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 19:05:12.136943 787 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/krb5cc' of type FILE
Sep 25 19:05:12 dist-test-slave-rn98 krb5kdc[6327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:12.137996 787 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 19:05:12 dist-test-slave-rn98 krb5kdc[6327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:12.139160 787 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 19:05:12 dist-test-slave-rn98 krb5kdc[6327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.193@KRBTEST.COM
I20260925 19:05:12.147361 6838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:12.139729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58180 (local address 127.0.196.193:38019)
0925 19:05:12.139784 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:12.139787 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:12.139844 (+ 57us) server_negotiation.cc:407] Connection header received
0925 19:05:12.139922 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:12.139922 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:12.139933 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:12.140009 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:12.140421 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:12.140979 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:12.141784 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:12.142021 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:12.142149 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:12.144560 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:12.144561 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:12.144561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:12.144569 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:12.146187 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:12.146537 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:12.146538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:12.146538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:12.146592 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:12.146857 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:12.146858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:12.146858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:12.146959 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:12.147042 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:12.147138 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 19:05:12.147203 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260925 19:05:12.148978 6838 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:12.148538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58182 (local address 127.0.196.193:38019)
0925 19:05:12.148575 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:12.148577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:12.148620 (+ 43us) server_negotiation.cc:407] Connection header received
0925 19:05:12.148669 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:12.148670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:12.148678 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:12.148729 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:12.148942 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58182: BlockingRecv error: recv got EOF from 127.0.0.1:58182 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 19:05:12.149142 787 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790363087086945-787-0/krb5kdc/krb5cc' of type FILE
Sep 25 19:05:12 dist-test-slave-rn98 krb5kdc[6327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:12.150285 787 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 19:05:12 dist-test-slave-rn98 krb5kdc[6327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:12.151280 787 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 19:05:12 dist-test-slave-rn98 krb5kdc[6327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.193@KRBTEST.COM
I20260925 19:05:12.159585 6838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:12.151799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58190 (local address 127.0.196.193:38019)
0925 19:05:12.151844 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:12.151846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:12.151859 (+ 13us) server_negotiation.cc:407] Connection header received
0925 19:05:12.151943 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:12.151943 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:12.151954 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:12.152012 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:12.152497 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:12.153117 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:12.153894 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:12.154126 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:12.154219 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:12.156686 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:12.156688 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:12.156688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:12.156698 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:12.158258 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:12.158682 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:12.158684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:12.158684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:12.158749 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:12.158972 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:12.158974 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:12.158974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:12.159113 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:12.159251 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:12.159386 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 19:05:12.159452 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:05:12.160288 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 6434
I20260925 19:05:12.165067 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 6568
I20260925 19:05:12.169871 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 6702
I20260925 19:05:12.174547 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 6340
2026-09-25T19:05:12Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1901 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:05:12 dist-test-slave-rn98 krb5kdc[6851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:05:12 dist-test-slave-rn98 krb5kdc[6851](info): set up 2 sockets
Sep 25 19:05:12 dist-test-slave-rn98 krb5kdc[6851](info): commencing operation
krb5kdc: starting...
W20260925 19:05:13.235080 787 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:05:13 dist-test-slave-rn98 krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:05:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:05:13Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.254.keytab.
Entry for principal oryx/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.254.keytab.
I20260925 19:05:13.357432 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:38655
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.254.keytab
--principal=oryx/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:33121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:38655
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:13.439728 6864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:13.439994 6864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:13.440032 6864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:13.441200 6864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:05:13.441269 6864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:13.441294 6864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:13.441322 6864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:05:13.441354 6864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:05:13.443423 6864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:33121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:38655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.254.keytab
--principal=oryx/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:38655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:13.443908 6864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:13.444207 6864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:13.451900 6869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:13.451973 6872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:13.451905 6864 server_base.cc:1034] running on GCE node
W20260925 19:05:13.452173 6870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:13.452533 6864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:13.452947 6864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:13.454099 6864 hybrid_clock.cc:648] HybridClock initialized: now 1790363113454080 us; error 43 us; skew 500 ppm
Sep 25 19:05:13 dist-test-slave-rn98 krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363113, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:13.457459 6864 init.cc:377] Logged in from keytab as oryx/127.0.196.254@KRBTEST.COM (short username oryx)
I20260925 19:05:13.458072 6864 webserver.cc:460] Webserver started at http://127.0.196.254:43499/ using document root <none> and password file <none>
I20260925 19:05:13.458367 6864 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:13.458441 6864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:13.458598 6864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:13.460233 6864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "2c9e241c502f4fed8aae1e80224e05b9"
format_stamp: "Formatted at 2026-09-25 19:05:13 on dist-test-slave-rn98"
server_key: "4a785cba5695a28d83867b75d9985169"
server_key_iv: "1952ca18530a8471efc4df3b83a988a5"
server_key_version: "encryptionkey@0"
I20260925 19:05:13.460695 6864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "2c9e241c502f4fed8aae1e80224e05b9"
format_stamp: "Formatted at 2026-09-25 19:05:13 on dist-test-slave-rn98"
server_key: "4a785cba5695a28d83867b75d9985169"
server_key_iv: "1952ca18530a8471efc4df3b83a988a5"
server_key_version: "encryptionkey@0"
I20260925 19:05:13.462579 6864 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260925 19:05:13.463562 6879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:13.463759 6864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:13.463858 6864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "2c9e241c502f4fed8aae1e80224e05b9"
format_stamp: "Formatted at 2026-09-25 19:05:13 on dist-test-slave-rn98"
server_key: "4a785cba5695a28d83867b75d9985169"
server_key_iv: "1952ca18530a8471efc4df3b83a988a5"
server_key_version: "encryptionkey@0"
I20260925 19:05:13.463935 6864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:13.487859 6864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:13.489944 6864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:13.490072 6864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:13.494485 6864 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:38655
I20260925 19:05:13.494516 6931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:38655 every 8 connection(s)
I20260925 19:05:13.494994 6864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:05:13.496083 6932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:13.498528 6932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9: Bootstrap starting.
I20260925 19:05:13.499210 6932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:13.499456 6932 log.cc:824] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:13.500283 6932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9: No bootstrap required, opened a new log
I20260925 19:05:13.501873 6932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } }
I20260925 19:05:13.502038 6932 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:13.502063 6932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c9e241c502f4fed8aae1e80224e05b9, State: Initialized, Role: FOLLOWER
I20260925 19:05:13.502224 6932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } }
I20260925 19:05:13.502313 6932 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:05:13.502337 6932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:05:13.502367 6932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:13.502733 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 6864
I20260925 19:05:13.502846 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:05:13.503046 787 external_mini_cluster.cc:1428] Setting key 605276907cbf88a7a9ac515ff3b27b43
I20260925 19:05:13.503700 6932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } }
I20260925 19:05:13.503824 6932 leader_election.cc:302] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c9e241c502f4fed8aae1e80224e05b9; no voters:
I20260925 19:05:13.504052 6932 leader_election.cc:288] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:05:13.504084 6937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:05:13.504238 6937 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [term 1 LEADER]: Becoming Leader. State: Replica: 2c9e241c502f4fed8aae1e80224e05b9, State: Running, Role: LEADER
I20260925 19:05:13.504362 6932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:05:13.504339 6937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } }
I20260925 19:05:13.504747 6937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c9e241c502f4fed8aae1e80224e05b9. Latest consensus state: current_term: 1 leader_uuid: "2c9e241c502f4fed8aae1e80224e05b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } }
I20260925 19:05:13.504830 6937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:13.505407 6944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:05:13.505548 6938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c9e241c502f4fed8aae1e80224e05b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } }
I20260925 19:05:13.505715 6938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:13.505929 6944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:05:13.507937 6944 catalog_manager.cc:1347] Generated new cluster ID: 1bd3a743ba4c4325831d9b5ff9980438
I20260925 19:05:13.508006 6944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:05:13 dist-test-slave-rn98 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM
I20260925 19:05:13.516530 6936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:13.503618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42166 (local address 127.0.196.254:38655)
0925 19:05:13.503835 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:13.503847 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:13.503910 (+ 63us) server_negotiation.cc:407] Connection header received
0925 19:05:13.504069 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:13.504077 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:13.504145 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:13.504306 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:13.505329 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:13.506781 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:13.507879 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:13.508487 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:13.508598 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:13.511556 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:13.511562 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:13.511562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:13.511577 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:13.514368 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:13.514844 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:13.514849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:13.514849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:13.514965 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:13.515139 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:13.515139 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:13.515140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:13.515450 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:13.515605 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:13.516115 (+ 510us) server_negotiation.cc:299] Negotiation successful
0925 19:05:13.516257 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":129,"threads_started":1}
I20260925 19:05:13.529629 6944 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:05:13.530429 6944 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:05:13.543947 6944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9: Generated new TSK 0
I20260925 19:05:13.544199 6944 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.193.keytab.
Entry for principal oryx/127.0.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.193.keytab.
I20260925 19:05:13.574982 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:0
--local_ip_for_outbound_sockets=127.0.196.193
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:38655
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.193.keytab
--principal=oryx/127.0.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:33121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:13.659741 6958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:13.659957 6958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:13.659984 6958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:13.661198 6958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:13.661244 6958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:13.661273 6958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:05:13.662810 6958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:33121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.193.keytab
--principal=oryx/127.0.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:38655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:13.663076 6958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:13.663319 6958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:13.668285 6963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:13.668325 6964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:13.668497 6958 server_base.cc:1034] running on GCE node
W20260925 19:05:13.668285 6966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:13.668807 6958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:13.669018 6958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:13.670173 6958 hybrid_clock.cc:648] HybridClock initialized: now 1790363113670162 us; error 27 us; skew 500 ppm
Sep 25 19:05:13 dist-test-slave-rn98 krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363113, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:13.676121 6958 init.cc:377] Logged in from keytab as oryx/127.0.196.193@KRBTEST.COM (short username oryx)
I20260925 19:05:13.676534 6958 webserver.cc:460] Webserver started at http://127.0.196.193:39641/ using document root <none> and password file <none>
I20260925 19:05:13.676726 6958 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:13.676767 6958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:13.676854 6958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:13.677976 6958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/data/instance:
uuid: "667b4a82651c4a26a717f37627d060d3"
format_stamp: "Formatted at 2026-09-25 19:05:13 on dist-test-slave-rn98"
server_key: "c4857ddca88f916322274d521f356018"
server_key_iv: "f52d075f598fa816ca8fa19f676bfe6c"
server_key_version: "encryptionkey@0"
I20260925 19:05:13.678301 6958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/wal/instance:
uuid: "667b4a82651c4a26a717f37627d060d3"
format_stamp: "Formatted at 2026-09-25 19:05:13 on dist-test-slave-rn98"
server_key: "c4857ddca88f916322274d521f356018"
server_key_iv: "f52d075f598fa816ca8fa19f676bfe6c"
server_key_version: "encryptionkey@0"
I20260925 19:05:13.679684 6958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:05:13.680413 6973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:13.680603 6958 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:13.680724 6958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "667b4a82651c4a26a717f37627d060d3"
format_stamp: "Formatted at 2026-09-25 19:05:13 on dist-test-slave-rn98"
server_key: "c4857ddca88f916322274d521f356018"
server_key_iv: "f52d075f598fa816ca8fa19f676bfe6c"
server_key_version: "encryptionkey@0"
I20260925 19:05:13.680797 6958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:13.692688 6958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:13.694733 6958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:13.694888 6958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:13.695116 6958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:13.695526 6958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:13.695580 6958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:13.695614 6958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:13.695648 6958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:13.701404 6958 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:34699
I20260925 19:05:13.701452 7086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:34699 every 8 connection(s)
I20260925 19:05:13.701784 6958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
Sep 25 19:05:13 dist-test-slave-rn98 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363113, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM
I20260925 19:05:13.710415 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 6958
I20260925 19:05:13.710559 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/wal/instance
I20260925 19:05:13.710776 787 external_mini_cluster.cc:1428] Setting key eeaf57f682a5bb49080d6778351f4a32
I20260925 19:05:13.711241 6936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:13.702489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:34589 (local address 127.0.196.254:38655)
0925 19:05:13.702538 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:13.702540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:13.702801 (+ 261us) server_negotiation.cc:407] Connection header received
0925 19:05:13.703340 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:13.703342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:13.703356 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:13.703409 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:13.704428 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:13.704929 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:13.706048 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:13.706252 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:13.706353 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:13.708451 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:13.708453 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:13.708453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:13.708461 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:13.709776 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:13.710184 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:13.710185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:13.710185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:13.710246 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:13.710498 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:13.710499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:13.710500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:13.710673 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:13.710736 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:13.711081 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 19:05:13.711177 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:05:13.711422 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:13.702461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.196.193:34589)
0925 19:05:13.702651 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:13.702699 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:13.703223 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:13.703482 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:13.703486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:13.704057 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:13.704325 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:13.704331 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:13.705020 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:13.705020 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:13.705940 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:13.705942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:13.706360 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:13.706362 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:13.706488 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:13.707006 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:13.707014 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:13.708353 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:13.709871 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:13.709879 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:13.709881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:13.710099 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:13.710313 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:13.710314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:13.710315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:13.710421 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:13.710727 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:13.710733 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:13.710901 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 19:05:13.711024 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 19:05:13.711199 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
I20260925 19:05:13.711907 7087 heartbeater.cc:347] Connected to a master server at 127.0.196.254:38655
I20260925 19:05:13.711988 7087 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:13.712139 7087 heartbeater.cc:511] Master 127.0.196.254:38655 requested a full tablet report, sending...
I20260925 19:05:13.712659 6896 ts_manager.cc:194] Registered new tserver with Master: 667b4a82651c4a26a717f37627d060d3 (127.0.196.193:34699)
I20260925 19:05:13.713353 6896 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.196.193@KRBTEST.COM'} at 127.0.196.193:34589
WARNING: no policy specified for oryx/127.0.196.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.194.keytab.
Entry for principal oryx/127.0.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.194.keytab.
I20260925 19:05:13.738124 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:0
--local_ip_for_outbound_sockets=127.0.196.194
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:38655
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.194.keytab
--principal=oryx/127.0.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:33121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:13.820760 7092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:13.820989 7092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:13.821015 7092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:13.822229 7092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:13.822276 7092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:13.822304 7092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:05:13.823899 7092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:33121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.194.keytab
--principal=oryx/127.0.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:38655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:13.824169 7092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:13.824445 7092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:13.829492 7100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:13.829474 7097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:13.829479 7098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:13.829735 7092 server_base.cc:1034] running on GCE node
I20260925 19:05:13.829933 7092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:13.830200 7092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:13.831384 7092 hybrid_clock.cc:648] HybridClock initialized: now 1790363113831353 us; error 50 us; skew 500 ppm
Sep 25 19:05:13 dist-test-slave-rn98 krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363113, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:13.833899 7092 init.cc:377] Logged in from keytab as oryx/127.0.196.194@KRBTEST.COM (short username oryx)
I20260925 19:05:13.834362 7092 webserver.cc:460] Webserver started at http://127.0.196.194:45633/ using document root <none> and password file <none>
I20260925 19:05:13.834555 7092 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:13.834596 7092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:13.834758 7092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:13.835788 7092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/data/instance:
uuid: "6a29d1c1c84644fea8558af4f2c9823f"
format_stamp: "Formatted at 2026-09-25 19:05:13 on dist-test-slave-rn98"
server_key: "1a89b8971c3c2bb3e0bab023ea45de19"
server_key_iv: "b23db239671011dac7de370a03ec7222"
server_key_version: "encryptionkey@0"
I20260925 19:05:13.836115 7092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/wal/instance:
uuid: "6a29d1c1c84644fea8558af4f2c9823f"
format_stamp: "Formatted at 2026-09-25 19:05:13 on dist-test-slave-rn98"
server_key: "1a89b8971c3c2bb3e0bab023ea45de19"
server_key_iv: "b23db239671011dac7de370a03ec7222"
server_key_version: "encryptionkey@0"
I20260925 19:05:13.837438 7092 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:05:13.838181 7107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:13.838352 7092 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:13.838433 7092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "6a29d1c1c84644fea8558af4f2c9823f"
format_stamp: "Formatted at 2026-09-25 19:05:13 on dist-test-slave-rn98"
server_key: "1a89b8971c3c2bb3e0bab023ea45de19"
server_key_iv: "b23db239671011dac7de370a03ec7222"
server_key_version: "encryptionkey@0"
I20260925 19:05:13.838490 7092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:13.856063 7092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:13.858201 7092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:13.858321 7092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:13.858515 7092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:13.858887 7092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:13.858929 7092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:13.858953 7092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:13.858965 7092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:13.864836 7092 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:43897
I20260925 19:05:13.864909 7220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:43897 every 8 connection(s)
I20260925 19:05:13.865254 7092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:05:13 dist-test-slave-rn98 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363113, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM
I20260925 19:05:13.873344 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 7092
I20260925 19:05:13.873435 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/wal/instance
I20260925 19:05:13.873731 787 external_mini_cluster.cc:1428] Setting key 30a392bd36160199ca909a09c06ff433
I20260925 19:05:13.875327 6936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:13.866042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:59985 (local address 127.0.196.254:38655)
0925 19:05:13.866128 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:13.866131 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:13.866307 (+ 176us) server_negotiation.cc:407] Connection header received
0925 19:05:13.866918 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:13.866919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:13.866931 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:13.866983 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:13.867991 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:13.868509 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:13.869618 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:13.869839 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:13.869911 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:13.872217 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:13.872219 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:13.872219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:13.872227 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:13.873609 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:13.874174 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:13.874175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:13.874176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:13.874234 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:13.874562 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:13.874563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:13.874564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:13.874729 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:13.874783 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:13.875193 (+ 410us) server_negotiation.cc:299] Negotiation successful
0925 19:05:13.875262 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 19:05:13.875658 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:13.866008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.196.194:59985)
0925 19:05:13.866184 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:13.866220 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:13.866803 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:13.867060 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:13.867064 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:13.867651 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:13.867866 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:13.867870 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:13.868607 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:13.868609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:13.869531 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:13.869533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:13.869935 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:13.869936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:13.870064 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:13.870527 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:13.870536 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:13.872120 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:13.873681 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:13.873689 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:13.873691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:13.874091 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:13.874299 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:13.874299 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:13.874300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:13.874483 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:13.874792 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:13.874798 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:13.875037 (+ 239us) client_negotiation.cc:769] Sending connection context
0925 19:05:13.875173 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 19:05:13.875357 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260925 19:05:13.876332 7221 heartbeater.cc:347] Connected to a master server at 127.0.196.254:38655
I20260925 19:05:13.876443 7221 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:13.876657 7221 heartbeater.cc:511] Master 127.0.196.254:38655 requested a full tablet report, sending...
I20260925 19:05:13.877020 6896 ts_manager.cc:194] Registered new tserver with Master: 6a29d1c1c84644fea8558af4f2c9823f (127.0.196.194:43897)
I20260925 19:05:13.877394 6896 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.196.194@KRBTEST.COM'} at 127.0.196.194:59985
WARNING: no policy specified for oryx/127.0.196.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.195.keytab.
Entry for principal oryx/127.0.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.195.keytab.
I20260925 19:05:13.901634 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:0
--local_ip_for_outbound_sockets=127.0.196.195
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:38655
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.195.keytab
--principal=oryx/127.0.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:33121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:13.983450 7226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:13.983687 7226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:13.983713 7226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:13.984983 7226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:13.985033 7226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:13.985062 7226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:05:13.986579 7226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:33121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.195.keytab
--principal=oryx/127.0.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:38655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:13.986799 7226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:13.986980 7226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:13.991926 7231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:13.991926 7232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:13.991972 7234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:13.992208 7226 server_base.cc:1034] running on GCE node
I20260925 19:05:13.992439 7226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:13.992703 7226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:13.993877 7226 hybrid_clock.cc:648] HybridClock initialized: now 1790363113993829 us; error 66 us; skew 500 ppm
Sep 25 19:05:13 dist-test-slave-rn98 krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363113, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:13.996721 7226 init.cc:377] Logged in from keytab as oryx/127.0.196.195@KRBTEST.COM (short username oryx)
I20260925 19:05:13.997185 7226 webserver.cc:460] Webserver started at http://127.0.196.195:40219/ using document root <none> and password file <none>
I20260925 19:05:13.997437 7226 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:13.997507 7226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:13.997642 7226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:13.998795 7226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/data/instance:
uuid: "4e5e4690e8104ca09a7592a858d2983c"
format_stamp: "Formatted at 2026-09-25 19:05:13 on dist-test-slave-rn98"
server_key: "9c1bcd94e08beffeca669bd646b07076"
server_key_iv: "5cb1dde3dec543c200ba33e8ce9562fa"
server_key_version: "encryptionkey@0"
I20260925 19:05:13.999171 7226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/wal/instance:
uuid: "4e5e4690e8104ca09a7592a858d2983c"
format_stamp: "Formatted at 2026-09-25 19:05:13 on dist-test-slave-rn98"
server_key: "9c1bcd94e08beffeca669bd646b07076"
server_key_iv: "5cb1dde3dec543c200ba33e8ce9562fa"
server_key_version: "encryptionkey@0"
I20260925 19:05:14.000581 7226 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 19:05:14.001390 7241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:14.001667 7226 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:05:14.001791 7226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "4e5e4690e8104ca09a7592a858d2983c"
format_stamp: "Formatted at 2026-09-25 19:05:13 on dist-test-slave-rn98"
server_key: "9c1bcd94e08beffeca669bd646b07076"
server_key_iv: "5cb1dde3dec543c200ba33e8ce9562fa"
server_key_version: "encryptionkey@0"
I20260925 19:05:14.001889 7226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:14.020131 7226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:14.022321 7226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:14.022495 7226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:14.022853 7226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:14.023197 7226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:14.023253 7226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:14.023294 7226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:14.023324 7226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:14.029140 7226 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:36681
I20260925 19:05:14.029237 7354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:36681 every 8 connection(s)
I20260925 19:05:14.029582 7226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
Sep 25 19:05:14 dist-test-slave-rn98 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363113, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM
I20260925 19:05:14.036937 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 7226
I20260925 19:05:14.037037 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/wal/instance
I20260925 19:05:14.037272 787 external_mini_cluster.cc:1428] Setting key b631e7becaa1c5d4e04cb1fc6c9a5a5c
I20260925 19:05:14.040539 6936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.030488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:49809 (local address 127.0.196.254:38655)
0925 19:05:14.030539 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:14.030541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:14.030798 (+ 257us) server_negotiation.cc:407] Connection header received
0925 19:05:14.031405 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:14.031407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:14.031421 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:14.031474 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:14.032533 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.033092 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.034212 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.034441 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.034529 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.037121 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:14.037123 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:14.037123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:14.037133 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:14.039285 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.039726 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.039727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.039727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.039784 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.039993 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.039993 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.039994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.040116 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:14.040167 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:14.040401 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 19:05:14.040469 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:05:14.040791 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.030430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.196.195:49809)
0925 19:05:14.030608 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:14.030686 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:14.031274 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:14.031550 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:14.031554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:14.032195 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:14.032405 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.032410 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.033202 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.033203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.034107 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.034109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.034534 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.034535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.034738 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.035307 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:14.035316 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:14.036973 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:14.039363 (+ 2390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.039369 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.039390 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.039663 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.039829 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.039830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.039830 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.039921 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.040171 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:14.040175 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:14.040291 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 19:05:14.040372 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 19:05:14.040515 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20260925 19:05:14.041379 7355 heartbeater.cc:347] Connected to a master server at 127.0.196.254:38655
I20260925 19:05:14.041495 7355 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:14.041687 7355 heartbeater.cc:511] Master 127.0.196.254:38655 requested a full tablet report, sending...
I20260925 19:05:14.042091 6896 ts_manager.cc:194] Registered new tserver with Master: 4e5e4690e8104ca09a7592a858d2983c (127.0.196.195:36681)
I20260925 19:05:14.042575 6896 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.196.195@KRBTEST.COM'} at 127.0.196.195:49809
I20260925 19:05:14.049587 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.196.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.196.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.196.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.253.keytab.
Entry for principal oryx/127.0.196.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.253.keytab.
I20260925 19:05:14.078824 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.253:36251
--webserver_interface=127.0.196.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.253.keytab
--principal=oryx/127.0.196.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:33121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:38655,127.0.196.253:36251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:14.163448 7360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:14.163715 7360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:14.163758 7360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:14.164965 7360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:05:14.165038 7360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:14.165063 7360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:14.165086 7360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:05:14.165117 7360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:05:14.166729 7360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:33121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:38655,127.0.196.253:36251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.253.keytab
--principal=oryx/127.0.196.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.253:36251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.196.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
W20260925 19:05:14.166890 7360 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.196.254:38655,127.0.196.253:36251'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 19:05:14.167058 7360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:14.167287 7360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:14.172845 7366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:14.172793 7368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:14.172947 7360 server_base.cc:1034] running on GCE node
W20260925 19:05:14.172793 7365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:14.173377 7360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:14.173672 7360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:14.174865 7360 hybrid_clock.cc:648] HybridClock initialized: now 1790363114174849 us; error 49 us; skew 500 ppm
Sep 25 19:05:14 dist-test-slave-rn98 krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363114, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:14.177568 7360 init.cc:377] Logged in from keytab as oryx/127.0.196.253@KRBTEST.COM (short username oryx)
I20260925 19:05:14.178081 7360 webserver.cc:460] Webserver started at http://127.0.196.253:33113/ using document root <none> and password file <none>
I20260925 19:05:14.178347 7360 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:14.178421 7360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:14.178587 7360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:14.179749 7360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-1/data/instance:
uuid: "fc40919e46b24cbebc864883e6aa8e29"
format_stamp: "Formatted at 2026-09-25 19:05:14 on dist-test-slave-rn98"
server_key: "036e67fe84ed7640864b48a7d8f9e8e9"
server_key_iv: "10998c9ef0cbfdec777559449402cbc6"
server_key_version: "encryptionkey@0"
I20260925 19:05:14.180142 7360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-1/wal/instance:
uuid: "fc40919e46b24cbebc864883e6aa8e29"
format_stamp: "Formatted at 2026-09-25 19:05:14 on dist-test-slave-rn98"
server_key: "036e67fe84ed7640864b48a7d8f9e8e9"
server_key_iv: "10998c9ef0cbfdec777559449402cbc6"
server_key_version: "encryptionkey@0"
I20260925 19:05:14.181561 7360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:05:14.182403 7375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:14.182704 7360 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:05:14.182829 7360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-1/wal
uuid: "fc40919e46b24cbebc864883e6aa8e29"
format_stamp: "Formatted at 2026-09-25 19:05:14 on dist-test-slave-rn98"
server_key: "036e67fe84ed7640864b48a7d8f9e8e9"
server_key_iv: "10998c9ef0cbfdec777559449402cbc6"
server_key_version: "encryptionkey@0"
I20260925 19:05:14.182938 7360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:14.209596 7360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:14.211859 7360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:14.212060 7360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:14.216701 7360 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.253:36251
I20260925 19:05:14.216737 7427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.253:36251 every 8 connection(s)
I20260925 19:05:14.217131 7360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-1/data/info.pb
I20260925 19:05:14.218111 7428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:14.219731 7428 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:05:14.224088 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 7360
Sep 25 19:05:14 dist-test-slave-rn98 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363114, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.253@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM
I20260925 19:05:14.229772 6936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.220443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42172 (local address 127.0.196.254:38655)
0925 19:05:14.220524 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:14.220527 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:14.220701 (+ 174us) server_negotiation.cc:407] Connection header received
0925 19:05:14.221147 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:14.221148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:14.221157 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:14.221212 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:14.221900 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.222381 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.223497 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.223694 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.223766 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.226553 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:14.226554 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:14.226555 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:14.226563 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:14.228211 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.228703 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.228705 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.228705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.228768 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.229059 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.229060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.229061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.229219 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:14.229297 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:14.229588 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 19:05:14.229682 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 19:05:14.229960 7430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.220391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.0.1:42172)
0925 19:05:14.220571 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:14.220609 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:14.221065 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:14.221328 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:14.221332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:14.221618 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:14.221820 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.221824 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.222468 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.222469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.223374 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.223376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.223784 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.223785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.223898 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.224495 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:14.224504 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:14.226439 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:14.228289 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.228294 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.228295 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.228628 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.228856 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.228857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.228858 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.228988 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.229299 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:14.229306 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:14.229451 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 19:05:14.229556 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 19:05:14.229712 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
Sep 25 19:05:14 dist-test-slave-rn98 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.196.253@KRBTEST.COM
I20260925 19:05:14.230721 7428 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:05:14.234246 7432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.224658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41890 (local address 127.0.196.253:36251)
0925 19:05:14.224854 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:14.224859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:14.224887 (+ 28us) server_negotiation.cc:407] Connection header received
0925 19:05:14.224909 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:14.224911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:14.224934 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:14.225010 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:14.225844 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.226887 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.228057 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.228451 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.228540 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.231088 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:14.231096 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:14.231096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:14.231103 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:14.232935 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.233352 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.233355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.233355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.233418 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.233610 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.233611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.233612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.233782 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:14.233949 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:14.234045 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 19:05:14.234124 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":116,"threads_started":1}
Sep 25 19:05:14 dist-test-slave-rn98 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363114, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.253@KRBTEST.COM for oryx/127.0.196.253@KRBTEST.COM
I20260925 19:05:14.238340 7430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.231057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:36251 (local address 127.0.0.1:41902)
0925 19:05:14.231100 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:14.231112 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:14.231146 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:14.231414 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:14.231415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:14.231577 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:14.231640 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.231642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.232605 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.232606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.233165 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.233167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.233654 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.233655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.233709 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.234151 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:14.234155 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:14.235569 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:14.237485 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.237486 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.237486 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.237670 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.237844 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.237845 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.237845 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.237894 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.238095 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:14.238095 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:14.238129 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 19:05:14.238179 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 19:05:14.238255 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 19:05:14.238480 7433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.231094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41902 (local address 127.0.196.253:36251)
0925 19:05:14.231274 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:14.231277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:14.231285 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:05:14.231305 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:14.231305 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:14.231317 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:14.231389 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:14.231714 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.232518 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.233251 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.233557 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.233646 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.235641 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:14.235642 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:14.235642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:14.235648 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:14.237412 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.237735 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.237736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.237736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.237786 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.237945 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.237946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.237946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.238042 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:14.238090 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:14.238219 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 19:05:14.238302 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":160,"thread_start_us":107,"threads_started":1}
I20260925 19:05:14.239709 7428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29: Bootstrap starting.
I20260925 19:05:14.240288 7428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:14.240505 7428 log.cc:824] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:14.241328 7428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29: No bootstrap required, opened a new log
I20260925 19:05:14.242899 7428 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } }
I20260925 19:05:14.243022 7428 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:14.243044 7428 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc40919e46b24cbebc864883e6aa8e29, State: Initialized, Role: FOLLOWER
I20260925 19:05:14.243134 7428 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } }
I20260925 19:05:14.243395 7435 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } } }
I20260925 19:05:14.243481 7435 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:05:14.243489 7428 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 19:05:14.245246 7446 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:05:14.245317 7446 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:05:14.246997 7360 master_runner.cc:428] Detected that this master fc40919e46b24cbebc864883e6aa8e29 is joining an existing cluster
I20260925 19:05:14.247056 7360 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.196.253:36251
I20260925 19:05:14.254557 6936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.248282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42176 (local address 127.0.196.254:38655)
0925 19:05:14.248327 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:14.248329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:14.248430 (+ 101us) server_negotiation.cc:407] Connection header received
0925 19:05:14.248507 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:14.248508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:14.248518 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:14.248577 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:14.248899 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.249403 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.250168 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.250377 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.250452 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.251624 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:14.251626 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:14.251627 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:14.251637 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:14.253252 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.253707 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.253708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.253709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.253776 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.253983 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.253984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.253984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.254116 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:14.254191 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:14.254365 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 19:05:14.254450 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:05:14.254606 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.248095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.0.1:42176)
0925 19:05:14.248352 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:14.248361 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:14.248445 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:14.248583 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:14.248584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:14.248761 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:14.248815 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.248817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.249489 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.249490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.250080 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.250081 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.250487 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.250489 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.250553 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.251098 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:14.251104 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:14.251545 (+ 441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:14.253344 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.253345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.253345 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.253590 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.253853 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.253854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.253854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.253911 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.254181 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:14.254182 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:14.254251 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 19:05:14.254337 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 19:05:14.254455 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":11,"thread_start_us":104,"threads_started":1}
I20260925 19:05:14.255390 7433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.248133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41912 (local address 127.0.196.253:36251)
0925 19:05:14.248179 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:14.248181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:14.248330 (+ 149us) server_negotiation.cc:407] Connection header received
0925 19:05:14.248422 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:14.248422 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:14.248431 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:14.248459 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:14.248899 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.249641 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.250725 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.251051 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.251090 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.252374 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:14.252375 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:14.252376 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:14.252381 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:14.254126 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.254518 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.254519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.254519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.254581 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.254811 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.254812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.254812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.254969 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:14.255019 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:14.255232 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 19:05:14.255307 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:05:14.255643 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.248090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:36251 (local address 127.0.0.1:41912)
0925 19:05:14.248261 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:14.248273 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:14.248355 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:14.248513 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:14.248514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:14.248731 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:14.248814 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.248817 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.249730 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.249731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.250602 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.250604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.251169 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.251170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.251228 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.251912 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:14.251917 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:14.252283 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:14.254210 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.254211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.254211 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.254445 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.254681 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.254682 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.254682 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.254740 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.255064 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:14.255064 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:14.255106 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 19:05:14.255459 (+ 353us) client_negotiation.cc:240] Negotiation successful
0925 19:05:14.255545 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":1,"thread_start_us":168,"threads_started":1}
Sep 25 19:05:14 dist-test-slave-rn98 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363113, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.254@KRBTEST.COM for oryx/127.0.196.253@KRBTEST.COM
I20260925 19:05:14.266120 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.257596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:36251 (local address 127.0.0.1:41922)
0925 19:05:14.258195 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:14.258234 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:14.258301 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:14.258531 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:14.258538 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:14.258930 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:14.258987 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.258990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.259913 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.259914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.260531 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.260534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.261002 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.261003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.261095 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.261662 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:14.261667 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:14.263210 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:14.265000 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.265007 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.265009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.265237 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.265437 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.265438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.265438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.265502 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.265773 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:14.265779 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:14.265828 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 19:05:14.265898 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 19:05:14.266000 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":87,"threads_started":1}
I20260925 19:05:14.266131 7433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.257643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41922 (local address 127.0.196.253:36251)
0925 19:05:14.257680 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:14.257681 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:14.258285 (+ 604us) server_negotiation.cc:407] Connection header received
0925 19:05:14.258360 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:14.258361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:14.258371 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:14.258435 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:14.259072 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.259809 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.260620 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.260925 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.260996 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.263304 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:14.263306 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:14.263306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:14.263314 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:14.264883 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.265298 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.265299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.265305 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.265368 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.265568 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.265569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.265569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.265709 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:14.265777 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:14.265943 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 19:05:14.266023 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:05:14.266901 6896 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.196.253:36251: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29"
member_type: NON_VOTER
last_known_addr {
host: "127.0.196.253"
port: 36251
}
attrs {
promote: true
}
}
dest_uuid: "2c9e241c502f4fed8aae1e80224e05b9"
cas_config_opid_index: -1
I20260925 19:05:14.267262 6896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: NON_VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: true } }
I20260925 19:05:14.267813 7456 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER fc40919e46b24cbebc864883e6aa8e29 (127.0.196.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: NON_VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: true } } }
I20260925 19:05:14.268229 7402 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:14.268975 7456 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.196.253:36251
I20260925 19:05:14.269187 7402 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 1 FOLLOWER]: Refusing update from remote peer 2c9e241c502f4fed8aae1e80224e05b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 19:05:14.269259 7360 master.cc:537] Master@127.0.196.253:36251 shutting down...
I20260925 19:05:14.269575 7456 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: NON_VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 19:05:14.270308 6883 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 19:05:14.270804 6883 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 -> Peer fc40919e46b24cbebc864883e6aa8e29 (127.0.196.253:36251): Couldn't send request to peer fc40919e46b24cbebc864883e6aa8e29. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 19:05:14.271592 7455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2c9e241c502f4fed8aae1e80224e05b9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: NON_VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: true } } }
I20260925 19:05:14.271744 7455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:14.272425 7360 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:05:14.272550 7360 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:05:14.272607 7360 tablet_replica.cc:330] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29: stopping tablet replica
I20260925 19:05:14.285719 7360 master.cc:552] Master@127.0.196.253:36251 shutdown complete.
I20260925 19:05:14.285776 7360 master_runner.cc:305] Clearing existing system tablet
I20260925 19:05:14.285920 7360 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:14.285976 7360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:14.286748 7360 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 19:05:14.287292 7460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:14.287418 7360 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:14.287485 7360 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:14.287508 7360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-1/wal
uuid: "fc40919e46b24cbebc864883e6aa8e29"
format_stamp: "Formatted at 2026-09-25 19:05:14 on dist-test-slave-rn98"
server_key: "036e67fe84ed7640864b48a7d8f9e8e9"
server_key_iv: "10998c9ef0cbfdec777559449402cbc6"
server_key_version: "encryptionkey@0"
I20260925 19:05:14.288228 7360 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:05:14.290072 7360 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 19:05:14.290160 7360 log.cc:1196] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 19:05:14.290416 7360 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29: Deleting consensus metadata
I20260925 19:05:14.290584 7360 master_runner.cc:315] Copying system tablet from 127.0.196.254:38655
I20260925 19:05:14.291317 7360 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29: tablet copy: Beginning tablet copy session from remote peer at address 127.0.196.254:38655
I20260925 19:05:14.298204 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.291682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.0.1:42186)
0925 19:05:14.291822 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:14.291834 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:14.291933 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:14.292110 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:14.292111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:14.292286 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:14.292338 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.292340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.293051 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.293053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.293623 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.293625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.294036 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.294037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.294110 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.294762 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:14.294766 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:14.295152 (+ 386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:14.297031 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.297039 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.297039 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.297238 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.297473 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.297474 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.297474 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.297547 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.297822 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:14.297823 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:14.297874 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 19:05:14.297964 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 19:05:14.298084 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260925 19:05:14.298187 6936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.291762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42186 (local address 127.0.196.254:38655)
0925 19:05:14.291819 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:14.291821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:14.291915 (+ 94us) server_negotiation.cc:407] Connection header received
0925 19:05:14.292004 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:14.292005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:14.292015 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:14.292070 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:14.292431 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.292947 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.293723 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.293938 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.294029 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.295262 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:14.295263 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:14.295264 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:14.295273 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:14.296918 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.297333 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.297334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.297335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.297404 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.297617 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.297618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.297618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.297752 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:14.297830 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:14.297991 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 19:05:14.298082 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 19:05:14.298836 6916 tablet_copy_service.cc:140] P 2c9e241c502f4fed8aae1e80224e05b9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fc40919e46b24cbebc864883e6aa8e29 ({username='oryx', principal='oryx/127.0.196.253@KRBTEST.COM'} at 127.0.0.1:42186)
I20260925 19:05:14.298933 6916 tablet_copy_service.cc:161] P 2c9e241c502f4fed8aae1e80224e05b9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fc40919e46b24cbebc864883e6aa8e29 at {username='oryx', principal='oryx/127.0.196.253@KRBTEST.COM'} at 127.0.0.1:42186: session id = fc40919e46b24cbebc864883e6aa8e29-00000000000000000000000000000000
I20260925 19:05:14.299621 6916 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 19:05:14.300359 7360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 19:05:14.301402 7360 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29: tablet copy: Starting download of 0 data blocks...
I20260925 19:05:14.301577 7360 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29: tablet copy: Starting download of 1 WAL segments...
I20260925 19:05:14.302587 7360 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 19:05:14.303505 6916 tablet_copy_service.cc:342] P 2c9e241c502f4fed8aae1e80224e05b9: Request end of tablet copy session fc40919e46b24cbebc864883e6aa8e29-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.196.253@KRBTEST.COM'} at 127.0.0.1:42186
I20260925 19:05:14.303575 6916 tablet_copy_service.cc:433] P 2c9e241c502f4fed8aae1e80224e05b9: ending tablet copy session fc40919e46b24cbebc864883e6aa8e29-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fc40919e46b24cbebc864883e6aa8e29
I20260925 19:05:14.306581 7360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:14.308072 7470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:14.308125 7467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:14.308279 7468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:14.310854 7360 server_base.cc:1034] running on GCE node
I20260925 19:05:14.310998 7360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:14.311194 7360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:14.312351 7360 hybrid_clock.cc:648] HybridClock initialized: now 1790363114312358 us; error 44 us; skew 500 ppm
Sep 25 19:05:14 dist-test-slave-rn98 krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363114, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:14.314008 7360 init.cc:377] Logged in from keytab as oryx/127.0.196.253@KRBTEST.COM (short username oryx)
I20260925 19:05:14.314368 7360 webserver.cc:460] Webserver started at http://127.0.196.253:38063/ using document root <none> and password file <none>
I20260925 19:05:14.314502 7360 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:14.314558 7360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:14.315269 7360 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:14.315864 7477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:14.316043 7360 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:05:14.316107 7360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-1/wal
uuid: "fc40919e46b24cbebc864883e6aa8e29"
format_stamp: "Formatted at 2026-09-25 19:05:14 on dist-test-slave-rn98"
server_key: "036e67fe84ed7640864b48a7d8f9e8e9"
server_key_iv: "10998c9ef0cbfdec777559449402cbc6"
server_key_version: "encryptionkey@0"
I20260925 19:05:14.316176 7360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:14.351907 7360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:14.353710 7360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:14.357020 7360 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.253:36251
I20260925 19:05:14.357052 7529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.253:36251 every 8 connection(s)
I20260925 19:05:14.357384 7360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-1/data/info.pb
I20260925 19:05:14.358168 7530 sys_catalog.cc:263] Verifying existing consensus state
I20260925 19:05:14.358594 7530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29: Bootstrap starting.
I20260925 19:05:14.360980 7530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:05:14.361212 7530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29: Bootstrap complete.
I20260925 19:05:14.361428 7530 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: NON_VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: true } }
I20260925 19:05:14.361497 7530 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fc40919e46b24cbebc864883e6aa8e29, State: Initialized, Role: LEARNER
I20260925 19:05:14.361524 7530 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: NON_VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: true } }
I20260925 19:05:14.361714 7530 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:05:14.361727 7533 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: NON_VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: true } } }
I20260925 19:05:14.361829 7533 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [sys.catalog]: This master's current role is: LEARNER
I20260925 19:05:14.363129 7544 catalog_manager.cc:1259] Loaded cluster ID: 1bd3a743ba4c4325831d9b5ff9980438
I20260925 19:05:14.363181 7544 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29: loading cluster ID for follower catalog manager: success
I20260925 19:05:14.364284 7544 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29: acquiring CA information for follower catalog manager: success
I20260925 19:05:14.364585 7544 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 19:05:14.393541 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.386682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41934 (local address 127.0.196.253:36251)
0925 19:05:14.386834 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:14.386836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:14.386843 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:05:14.386866 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:14.386866 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:14.386876 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:14.386950 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:14.387572 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.388277 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.389044 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.389258 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.389343 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.390683 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:14.390685 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:14.390685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:14.390693 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:14.392247 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.392653 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.392655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.392655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.392717 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.392929 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.392930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.392931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.393074 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:14.393193 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:14.393303 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 19:05:14.393391 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for oryx/127.0.196.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.196.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.196.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.252.keytab.
Entry for principal oryx/127.0.196.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.252.keytab.
I20260925 19:05:14.421908 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.252:33081
--webserver_interface=127.0.196.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.252.keytab
--principal=oryx/127.0.196.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:33121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:38655,127.0.196.253:36251,127.0.196.252:33081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:14.507251 7548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:14.507475 7548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:14.507500 7548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:14.508800 7548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:05:14.508854 7548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:14.508870 7548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:14.508909 7548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:05:14.508939 7548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:05:14.510551 7548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:33121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:38655,127.0.196.253:36251,127.0.196.252:33081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.252.keytab
--principal=oryx/127.0.196.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.252:33081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.196.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:14.510881 7548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:14.511149 7548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:14.516788 7553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:14.516988 7548 server_base.cc:1034] running on GCE node
W20260925 19:05:14.516791 7556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:14.516798 7554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:14.517393 7548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:14.517735 7548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:14.518900 7548 hybrid_clock.cc:648] HybridClock initialized: now 1790363114518892 us; error 38 us; skew 500 ppm
Sep 25 19:05:14 dist-test-slave-rn98 krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363114, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:14.521466 7548 init.cc:377] Logged in from keytab as oryx/127.0.196.252@KRBTEST.COM (short username oryx)
I20260925 19:05:14.521979 7548 webserver.cc:460] Webserver started at http://127.0.196.252:33603/ using document root <none> and password file <none>
I20260925 19:05:14.522228 7548 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:14.522296 7548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:14.522455 7548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:14.523617 7548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-2/data/instance:
uuid: "dfdd31dd8df548429c1f60d15fc974fa"
format_stamp: "Formatted at 2026-09-25 19:05:14 on dist-test-slave-rn98"
server_key: "6a6ea386e1803768678faa330c09647f"
server_key_iv: "84fd4dfeff24ab39df15d652b57b4d94"
server_key_version: "encryptionkey@0"
I20260925 19:05:14.524011 7548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-2/wal/instance:
uuid: "dfdd31dd8df548429c1f60d15fc974fa"
format_stamp: "Formatted at 2026-09-25 19:05:14 on dist-test-slave-rn98"
server_key: "6a6ea386e1803768678faa330c09647f"
server_key_iv: "84fd4dfeff24ab39df15d652b57b4d94"
server_key_version: "encryptionkey@0"
I20260925 19:05:14.525434 7548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:05:14.526238 7563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:14.526412 7548 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:14.526510 7548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-2/wal
uuid: "dfdd31dd8df548429c1f60d15fc974fa"
format_stamp: "Formatted at 2026-09-25 19:05:14 on dist-test-slave-rn98"
server_key: "6a6ea386e1803768678faa330c09647f"
server_key_iv: "84fd4dfeff24ab39df15d652b57b4d94"
server_key_version: "encryptionkey@0"
I20260925 19:05:14.526593 7548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:14.541110 7548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:14.543485 7548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:14.543655 7548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:14.547961 7548 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.252:33081
I20260925 19:05:14.548000 7615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.252:33081 every 8 connection(s)
I20260925 19:05:14.548375 7548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-2/data/info.pb
I20260925 19:05:14.549381 7616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:14.551038 7616 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 19:05:14 dist-test-slave-rn98 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363114, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.252@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM
I20260925 19:05:14.557257 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 7548
I20260925 19:05:14.560814 6936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.551832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42188 (local address 127.0.196.254:38655)
0925 19:05:14.551895 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:14.551902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:14.552042 (+ 140us) server_negotiation.cc:407] Connection header received
0925 19:05:14.552499 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:14.552500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:14.552508 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:14.552570 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:14.553196 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.553684 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.554714 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.554911 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.554969 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.557238 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:14.557239 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:14.557239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:14.557246 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:14.559257 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.559793 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.559795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.559795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.559853 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.560138 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.560138 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.560139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.560242 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:14.560328 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:14.560630 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 19:05:14.560713 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 19:05:14.560932 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.551736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.0.1:42188)
0925 19:05:14.551907 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:14.551950 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:14.552415 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:14.552656 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:14.552659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:14.552917 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:14.553119 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.553123 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.553763 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.553764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.554564 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.554565 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.555004 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.555005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.555110 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.555682 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:14.555691 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:14.557140 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:14.559369 (+ 2229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.559375 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.559378 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.559712 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.559924 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.559925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.559925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.560074 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.560340 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:14.560344 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:14.560474 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 19:05:14.560572 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 19:05:14.560706 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
I20260925 19:05:14.561739 7616 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 19:05:14 dist-test-slave-rn98 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.196.252@KRBTEST.COM
Sep 25 19:05:14 dist-test-slave-rn98 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363114, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.252@KRBTEST.COM for oryx/127.0.196.253@KRBTEST.COM
I20260925 19:05:14.568012 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.557831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59232 (local address 127.0.196.252:33081)
0925 19:05:14.558048 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:14.558055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:14.558085 (+ 30us) server_negotiation.cc:407] Connection header received
0925 19:05:14.558110 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:14.558112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:14.558138 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:14.558216 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:14.558932 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.560204 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.561169 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.561664 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.561744 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.564747 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:14.564758 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:14.564758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:14.564765 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:14.566753 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.567127 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.567131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.567131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.567196 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.567395 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.567396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.567397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.567556 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:14.567746 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:14.567791 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 19:05:14.567889 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":126,"threads_started":1}
I20260925 19:05:14.569718 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.562034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:36251 (local address 127.0.0.1:41942)
0925 19:05:14.562191 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:14.562201 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:14.562245 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:14.562395 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:14.562396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:14.562673 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:14.562736 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.562738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.563554 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.563555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.564215 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.564217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.564658 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.564659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.564732 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.565171 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:14.565176 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:14.566777 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:14.568582 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.568584 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.568584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.568793 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.569071 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.569072 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.569072 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.569129 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.569381 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:14.569381 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:14.569425 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 19:05:14.569514 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 19:05:14.569618 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260925 19:05:14.569717 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.562111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41942 (local address 127.0.196.253:36251)
0925 19:05:14.562158 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:14.562160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:14.562237 (+ 77us) server_negotiation.cc:407] Connection header received
0925 19:05:14.562329 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:14.562330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:14.562343 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:14.562388 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:14.562829 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.563425 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.564306 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.564569 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.564659 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.566887 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:14.566889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:14.566889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:14.566897 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:14.568498 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.568851 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.568853 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.568853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.568953 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.569198 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.569199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.569200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.569313 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:14.569389 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:14.569546 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 19:05:14.569628 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:05:14.570178 7616 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.252" port: 33081 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 19:05:14 dist-test-slave-rn98 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363114, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.252@KRBTEST.COM for oryx/127.0.196.252@KRBTEST.COM
I20260925 19:05:14.577092 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.570442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:33081 (local address 127.0.0.1:59236)
0925 19:05:14.570494 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:14.570504 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:14.570542 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:14.570677 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:14.570678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:14.570812 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:14.570842 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.570843 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.571503 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.571504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.571978 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.571979 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.572346 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.572347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.572393 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.572803 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:14.572807 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:14.573976 (+ 1169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:14.576001 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.576002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.576003 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.576174 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.576380 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.576381 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.576382 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.576436 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.576707 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:14.576708 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:14.576748 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 19:05:14.576832 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 19:05:14.576987 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":70}
I20260925 19:05:14.577152 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.570503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59236 (local address 127.0.196.252:33081)
0925 19:05:14.570550 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:14.570551 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:14.570561 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:05:14.570584 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:14.570585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:14.570594 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:14.570671 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:14.570935 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.571409 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.572059 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.572263 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.572348 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.574066 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:14.574067 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:14.574068 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:14.574075 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:14.575882 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.576257 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.576258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.576258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.576310 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.576514 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.576515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.576516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.576635 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:14.576706 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:14.576854 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 19:05:14.576940 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 19:05:14.578747 7616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa: Bootstrap starting.
I20260925 19:05:14.579701 7616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:14.579981 7616 log.cc:824] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:14.580761 7616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa: No bootstrap required, opened a new log
I20260925 19:05:14.582832 7616 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 33081 } }
I20260925 19:05:14.583051 7616 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:14.583117 7616 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfdd31dd8df548429c1f60d15fc974fa, State: Initialized, Role: FOLLOWER
I20260925 19:05:14.583293 7616 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 33081 } }
I20260925 19:05:14.583622 7621 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 33081 } } }
I20260925 19:05:14.583686 7616 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:05:14.583719 7621 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [sys.catalog]: This master's current role is: FOLLOWER
W20260925 19:05:14.585538 7632 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:05:14.585613 7632 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:05:14.588114 7548 master_runner.cc:428] Detected that this master dfdd31dd8df548429c1f60d15fc974fa is joining an existing cluster
I20260925 19:05:14.588187 7548 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.196.252:33081
I20260925 19:05:14.596364 6936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.589306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42190 (local address 127.0.196.254:38655)
0925 19:05:14.589363 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:14.589365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:14.589611 (+ 246us) server_negotiation.cc:407] Connection header received
0925 19:05:14.589742 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:14.589743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:14.589756 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:14.589813 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:14.590439 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.591032 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.592161 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.592395 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.592428 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.593599 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:14.593602 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:14.593602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:14.593610 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:14.595112 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.595486 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.595487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.595487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.595558 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.595760 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.595761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.595762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.595929 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:14.595993 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:14.596168 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 19:05:14.596255 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:05:14.596391 7639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.589238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.0.1:42190)
0925 19:05:14.589547 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:14.589559 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:14.589623 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:14.589876 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:14.589876 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:14.590205 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:14.590333 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.590336 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.591123 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.591124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.592057 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.592059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.592505 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.592505 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.592560 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.593169 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:14.593173 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:14.593500 (+ 327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:14.595200 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.595201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.595202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.595401 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.595628 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.595629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.595629 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.595692 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.595995 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:14.595996 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:14.596053 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 19:05:14.596141 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 19:05:14.596258 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":20,"spinlock_wait_cycles":12544,"thread_start_us":133,"threads_started":1}
Sep 25 19:05:14 dist-test-slave-rn98 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363113, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.254@KRBTEST.COM for oryx/127.0.196.252@KRBTEST.COM
I20260925 19:05:14.614828 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.599110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:33081 (local address 127.0.0.1:59260)
0925 19:05:14.599192 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:14.599203 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:14.599240 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:14.599628 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:14.599628 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:14.599762 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:14.599801 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.599803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.600771 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.600772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.601297 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.601299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.601849 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.601850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.601909 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.602443 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:14.602448 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:14.603896 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:14.613445 (+ 9549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.613447 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.613448 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.613742 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.614011 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.614012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.614012 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.614091 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.614426 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:14.614428 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:14.614479 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 19:05:14.614527 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 19:05:14.614694 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 19:05:14.615312 7641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.599325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59260 (local address 127.0.196.252:33081)
0925 19:05:14.599473 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:14.599475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:14.599482 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:05:14.599517 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:14.599518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:14.599539 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:14.599611 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:14.599887 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.600681 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.601404 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.601728 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.601802 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.604366 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:14.604368 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:14.604369 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:14.604376 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:14.613320 (+ 8944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.613839 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.613841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.613841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.613917 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.614175 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.614175 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.614176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.614337 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:14.615022 (+ 685us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:14.615060 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 19:05:14.615174 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
I20260925 19:05:14.615595 7639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.589565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:33081 (local address 127.0.0.1:59244)
0925 19:05:14.596755 (+ 7190us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:14.596770 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:14.596849 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:14.596965 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:14.596966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:14.597116 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:14.597203 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.597205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.598005 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.598006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.598803 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.598806 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.599281 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.599282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.599373 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.611453 (+ 12080us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:14.611461 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:14.612071 (+ 610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:14.614251 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.614252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.614253 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.614447 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.614657 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.614658 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.614658 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.614710 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.615254 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:14.615254 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:14.615296 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 19:05:14.615357 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 19:05:14.615453 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7174,"thread_start_us":70,"threads_started":1}
I20260925 19:05:14.615940 6896 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.196.252:33081: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa"
member_type: NON_VOTER
last_known_addr {
host: "127.0.196.252"
port: 33081
}
attrs {
promote: true
}
}
dest_uuid: "2c9e241c502f4fed8aae1e80224e05b9"
cas_config_opid_index: 5
I20260925 19:05:14.616210 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.589435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41956 (local address 127.0.196.253:36251)
0925 19:05:14.589480 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:14.589482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:14.589733 (+ 251us) server_negotiation.cc:407] Connection header received
0925 19:05:14.589799 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:14.589800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:14.589811 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:14.589864 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:14.590422 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.591229 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.592217 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.592519 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.592604 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.612837 (+ 20233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:14.612839 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:14.612840 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:14.612850 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:14.614881 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.615313 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.615315 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.615315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.615376 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.615623 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.615624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.615624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.615766 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:14.615811 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:14.616040 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 19:05:14.616133 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:05:14.616216 6896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: NON_VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: true } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: NON_VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: true } }
I20260925 19:05:14.616428 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.589538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59244 (local address 127.0.196.252:33081)
0925 19:05:14.589575 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:14.589577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:14.596824 (+ 7247us) server_negotiation.cc:407] Connection header received
0925 19:05:14.596905 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:14.596905 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:14.596914 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:14.596954 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:14.597280 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.597915 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.598891 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.599200 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.599271 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.612203 (+ 12932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:14.612205 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:14.612205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:14.612213 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:14.614157 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.614512 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.614513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.614513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.614570 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.614766 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.614767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.614767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.614874 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:14.614929 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:14.616268 (+ 1339us) server_negotiation.cc:299] Negotiation successful
0925 19:05:14.616346 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:05:14.616598 7640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.589240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:36251 (local address 127.0.0.1:41956)
0925 19:05:14.589670 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:14.589680 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:14.589745 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:14.589864 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:14.589865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:14.590201 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:14.590333 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.590336 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.591317 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.591318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.592132 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.592134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.592610 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.592611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.592674 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.611484 (+ 18810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:14.611487 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:14.612709 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:14.614975 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.614977 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.614977 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.615225 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.615454 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.615455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.615455 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.615551 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.615861 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:14.615862 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:14.615926 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 19:05:14.616353 (+ 427us) client_negotiation.cc:240] Negotiation successful
0925 19:05:14.616487 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":21,"thread_start_us":101,"threads_started":1}
I20260925 19:05:14.616806 7455 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER dfdd31dd8df548429c1f60d15fc974fa (127.0.196.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: NON_VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: true } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: NON_VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: true } } }
I20260925 19:05:14.617827 7455 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.196.252:33081
I20260925 19:05:14.617910 7455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2c9e241c502f4fed8aae1e80224e05b9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: NON_VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: true } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: NON_VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: true } } }
I20260925 19:05:14.617990 7455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:14.618311 7548 master.cc:537] Master@127.0.196.252:33081 shutting down...
W20260925 19:05:14.618906 6881 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 -> Peer dfdd31dd8df548429c1f60d15fc974fa (127.0.196.252:33081): Couldn't send request to peer dfdd31dd8df548429c1f60d15fc974fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 19:05:14.624346 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.618129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:36251 (local address 127.0.0.1:41972)
0925 19:05:14.618185 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:14.618198 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:14.618246 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:14.618784 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:14.618786 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:14.618788 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:14.618839 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.618840 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.619766 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.619767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.621015 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.621018 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.623977 (+ 2959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.623979 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.624063 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.624099 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 19:05:14.624157 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 19:05:14.624241 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 19:05:14.624349 7548 raft_consensus.cc:2229] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [term 0 FOLLOWER]: Raft consensus shutting down.
I20260925 19:05:14.624480 7548 raft_consensus.cc:2258] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [term 0 FOLLOWER]: Raft consensus is shut down!
I20260925 19:05:14.624527 7548 tablet_replica.cc:330] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa: stopping tablet replica
I20260925 19:05:14.624663 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.618259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41972 (local address 127.0.196.253:36251)
0925 19:05:14.618544 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:14.618546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:14.618555 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:05:14.618594 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:14.618595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:14.618656 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:14.618729 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:14.618929 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.619678 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.622263 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.623063 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.623162 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.623286 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:14.624518 (+ 1232us) server_negotiation.cc:299] Negotiation successful
0925 19:05:14.624585 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265}
I20260925 19:05:14.624900 7504 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 1 LEARNER]: Refusing update from remote peer 2c9e241c502f4fed8aae1e80224e05b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 19:05:14.625159 7642 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: NON_VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 19:05:14.626019 7504 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER dfdd31dd8df548429c1f60d15fc974fa (127.0.196.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: NON_VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: true } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: NON_VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: true } } }
I20260925 19:05:14.627310 7644 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c9e241c502f4fed8aae1e80224e05b9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: NON_VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: true } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: NON_VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: true } } }
I20260925 19:05:14.627399 7644 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [sys.catalog]: This master's current role is: LEARNER
I20260925 19:05:14.627524 7533 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c9e241c502f4fed8aae1e80224e05b9. Latest consensus state: current_term: 1 leader_uuid: "2c9e241c502f4fed8aae1e80224e05b9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: NON_VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: true } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: NON_VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: true } } }
I20260925 19:05:14.627628 7533 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [sys.catalog]: This master's current role is: LEARNER
I20260925 19:05:14.627964 7455 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9: attempting to promote NON_VOTER fc40919e46b24cbebc864883e6aa8e29 to VOTER
I20260925 19:05:14.628202 7455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: NON_VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: true } }
I20260925 19:05:14.628621 7504 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 1 LEARNER]: Refusing update from remote peer 2c9e241c502f4fed8aae1e80224e05b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 19:05:14.628801 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.628573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:33081 (local address 127.0.0.1:59272)
0925 19:05:14.628610 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:14.628659 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260925 19:05:14.628875 7642 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 19:05:14.629112 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.629028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:33081 (local address 127.0.0.1:59282)
0925 19:05:14.629062 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:14.629085 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260925 19:05:14.629184 6881 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 -> Peer dfdd31dd8df548429c1f60d15fc974fa (127.0.196.252:33081): Couldn't send request to peer dfdd31dd8df548429c1f60d15fc974fa. Status: Network error: Client connection negotiation failed: client connection to 127.0.196.252:33081: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 19:05:14.629532 7456 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, fc40919e46b24cbebc864883e6aa8e29 (127.0.196.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: NON_VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: true } } }
I20260925 19:05:14.629655 7504 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, fc40919e46b24cbebc864883e6aa8e29 (127.0.196.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: NON_VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: true } } }
I20260925 19:05:14.630506 7533 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c9e241c502f4fed8aae1e80224e05b9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: NON_VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: true } } }
I20260925 19:05:14.630474 7456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c9e241c502f4fed8aae1e80224e05b9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: NON_VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: true } } }
I20260925 19:05:14.630574 7533 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:05:14.630589 7456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:14.630470 7642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2c9e241c502f4fed8aae1e80224e05b9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: NON_VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: true } } }
I20260925 19:05:14.630802 7642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:14.637784 7548 master.cc:552] Master@127.0.196.252:33081 shutdown complete.
I20260925 19:05:14.637837 7548 master_runner.cc:305] Clearing existing system tablet
I20260925 19:05:14.637984 7548 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:14.638022 7548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:14.638805 7548 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.002s sys 0.000s
I20260925 19:05:14.639285 7646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:14.639403 7548 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:14.639459 7548 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:14.639482 7548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-2/wal
uuid: "dfdd31dd8df548429c1f60d15fc974fa"
format_stamp: "Formatted at 2026-09-25 19:05:14 on dist-test-slave-rn98"
server_key: "6a6ea386e1803768678faa330c09647f"
server_key_iv: "84fd4dfeff24ab39df15d652b57b4d94"
server_key_version: "encryptionkey@0"
I20260925 19:05:14.640130 7548 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:05:14.641721 7548 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 19:05:14.641796 7548 log.cc:1196] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 19:05:14.642012 7548 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa: Deleting consensus metadata
I20260925 19:05:14.642184 7548 master_runner.cc:315] Copying system tablet from 127.0.196.254:38655
I20260925 19:05:14.642843 7548 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa: tablet copy: Beginning tablet copy session from remote peer at address 127.0.196.254:38655
I20260925 19:05:14.649145 6936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.643412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42198 (local address 127.0.196.254:38655)
0925 19:05:14.643474 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:14.643477 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:14.643484 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:05:14.643510 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:14.643511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:14.643520 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:14.643600 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:14.643939 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.644441 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.645184 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.645396 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.645485 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.646542 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:14.646543 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:14.646544 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:14.646569 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:14.647961 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.648322 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.648324 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.648324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.648391 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.648608 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.648609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.648610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.648731 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:14.648805 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:14.648975 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 19:05:14.649046 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 19:05:14.649375 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.643199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.0.1:42198)
0925 19:05:14.643323 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:14.643336 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:14.643406 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:14.643604 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:14.643605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:14.643793 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:14.643854 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.643856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.644527 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.644528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.645093 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.645094 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.645487 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.645488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.645556 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.646136 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:14.646142 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:14.646435 (+ 293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:14.648066 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.648067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.648068 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.648249 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.648468 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.648468 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.648469 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.648542 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.648807 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:14.648808 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:14.648853 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 19:05:14.648932 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 19:05:14.649054 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":83,"threads_started":1}
I20260925 19:05:14.649638 6916 tablet_copy_service.cc:140] P 2c9e241c502f4fed8aae1e80224e05b9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer dfdd31dd8df548429c1f60d15fc974fa ({username='oryx', principal='oryx/127.0.196.252@KRBTEST.COM'} at 127.0.0.1:42198)
I20260925 19:05:14.649699 6916 tablet_copy_service.cc:161] P 2c9e241c502f4fed8aae1e80224e05b9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer dfdd31dd8df548429c1f60d15fc974fa at {username='oryx', principal='oryx/127.0.196.252@KRBTEST.COM'} at 127.0.0.1:42198: session id = dfdd31dd8df548429c1f60d15fc974fa-00000000000000000000000000000000
I20260925 19:05:14.650197 6916 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 19:05:14.650774 7548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 19:05:14.652284 7548 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa: tablet copy: Starting download of 0 data blocks...
I20260925 19:05:14.652494 7548 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa: tablet copy: Starting download of 1 WAL segments...
I20260925 19:05:14.653424 7548 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 19:05:14.654502 6916 tablet_copy_service.cc:342] P 2c9e241c502f4fed8aae1e80224e05b9: Request end of tablet copy session dfdd31dd8df548429c1f60d15fc974fa-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.196.252@KRBTEST.COM'} at 127.0.0.1:42198
I20260925 19:05:14.654572 6916 tablet_copy_service.cc:433] P 2c9e241c502f4fed8aae1e80224e05b9: ending tablet copy session dfdd31dd8df548429c1f60d15fc974fa-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer dfdd31dd8df548429c1f60d15fc974fa
I20260925 19:05:14.657235 7548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:14.658483 7653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:14.658594 7656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:14.658675 7654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:14.661412 7548 server_base.cc:1034] running on GCE node
I20260925 19:05:14.661546 7548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:14.661700 7548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:14.662819 7548 hybrid_clock.cc:648] HybridClock initialized: now 1790363114662821 us; error 37 us; skew 500 ppm
Sep 25 19:05:14 dist-test-slave-rn98 krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363114, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:14.664248 7548 init.cc:377] Logged in from keytab as oryx/127.0.196.252@KRBTEST.COM (short username oryx)
I20260925 19:05:14.664556 7548 webserver.cc:460] Webserver started at http://127.0.196.252:40963/ using document root <none> and password file <none>
I20260925 19:05:14.664693 7548 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:14.664740 7548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:14.665452 7548 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:05:14.666049 7663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:14.666239 7548 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:14.666301 7548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-2/wal
uuid: "dfdd31dd8df548429c1f60d15fc974fa"
format_stamp: "Formatted at 2026-09-25 19:05:14 on dist-test-slave-rn98"
server_key: "6a6ea386e1803768678faa330c09647f"
server_key_iv: "84fd4dfeff24ab39df15d652b57b4d94"
server_key_version: "encryptionkey@0"
I20260925 19:05:14.666368 7548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:14.687798 7548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:14.689663 7548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:14.692885 7548 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.252:33081
I20260925 19:05:14.693225 7715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.252:33081 every 8 connection(s)
I20260925 19:05:14.693311 7548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/master-2/data/info.pb
I20260925 19:05:14.694115 7716 sys_catalog.cc:263] Verifying existing consensus state
I20260925 19:05:14.694552 7716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa: Bootstrap starting.
I20260925 19:05:14.697180 7716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:05:14.697425 7716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa: Bootstrap complete.
I20260925 19:05:14.697656 7716 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: NON_VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: true } }
I20260925 19:05:14.697774 7716 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dfdd31dd8df548429c1f60d15fc974fa, State: Initialized, Role: LEARNER
I20260925 19:05:14.697818 7716 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: NON_VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: true } }
I20260925 19:05:14.698026 7716 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:05:14.698014 7719 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: NON_VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: true } } }
I20260925 19:05:14.698110 7719 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [sys.catalog]: This master's current role is: LEARNER
I20260925 19:05:14.699992 7730 catalog_manager.cc:1259] Loaded cluster ID: 1bd3a743ba4c4325831d9b5ff9980438
I20260925 19:05:14.700067 7730 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa: loading cluster ID for follower catalog manager: success
I20260925 19:05:14.701005 7730 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa: acquiring CA information for follower catalog manager: success
I20260925 19:05:14.701366 7730 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 19:05:14.714265 7087 heartbeater.cc:503] Master 127.0.196.254:38655 was elected leader, sending a full tablet report...
I20260925 19:05:14.734648 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.727519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59310 (local address 127.0.196.252:33081)
0925 19:05:14.727688 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:14.727690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:14.727697 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:05:14.727721 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:14.727721 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:14.727731 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:14.727820 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:14.728404 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.729104 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.729894 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.730158 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.730240 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.731644 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:14.731646 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:14.731646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:14.731654 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:14.733121 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.733478 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.733479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.733479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.733544 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.733807 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.733808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.733808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.733959 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:14.734102 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:14.734346 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 19:05:14.734461 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":91,"threads_started":1}
I20260925 19:05:14.735358 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 6958
I20260925 19:05:14.741098 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:34699
--local_ip_for_outbound_sockets=127.0.196.193
--tserver_master_addrs=127.0.196.254:38655
--webserver_port=39641
--webserver_interface=127.0.196.193
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.193.keytab
--principal=oryx/127.0.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:33121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.196.254:38655,127.0.196.253:36251
--tserver_master_addrs=127.0.196.254:38655,127.0.196.253:36251,127.0.196.252:33081 with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:14.825579 7732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:14.825903 7732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:14.825946 7732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:14.827229 7732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:14.827306 7732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:14.827353 7732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:05:14.828967 7732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:33121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.193.keytab
--principal=oryx/127.0.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:34699
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=39641
--tserver_master_addrs=127.0.196.254:38655,127.0.196.253:36251,127.0.196.252:33081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:14.829268 7732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:14.829512 7732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:14.834873 7738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:14.834896 7737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:14.834887 7740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:14.834916 7732 server_base.cc:1034] running on GCE node
I20260925 19:05:14.835278 7732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:14.835487 7732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:14.836632 7732 hybrid_clock.cc:648] HybridClock initialized: now 1790363114836614 us; error 47 us; skew 500 ppm
Sep 25 19:05:14 dist-test-slave-rn98 krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363114, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:14.839171 7732 init.cc:377] Logged in from keytab as oryx/127.0.196.193@KRBTEST.COM (short username oryx)
I20260925 19:05:14.839655 7732 webserver.cc:460] Webserver started at http://127.0.196.193:39641/ using document root <none> and password file <none>
I20260925 19:05:14.839843 7732 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:14.839893 7732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:14.841259 7732 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:14.841832 7747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:14.842043 7732 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:05:14.842139 7732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "667b4a82651c4a26a717f37627d060d3"
format_stamp: "Formatted at 2026-09-25 19:05:13 on dist-test-slave-rn98"
server_key: "c4857ddca88f916322274d521f356018"
server_key_iv: "f52d075f598fa816ca8fa19f676bfe6c"
server_key_version: "encryptionkey@0"
I20260925 19:05:14.842536 7732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:14.852309 7732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:14.854339 7732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:14.854521 7732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:14.854830 7732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:14.855170 7732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:14.855204 7732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:14.855227 7732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:14.855275 7732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:14.861933 7732 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:34699
I20260925 19:05:14.862021 7860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:34699 every 8 connection(s)
I20260925 19:05:14.862447 7732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
I20260925 19:05:14.866370 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 7732
I20260925 19:05:14.866542 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 7092
Sep 25 19:05:14 dist-test-slave-rn98 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363114, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM
Sep 25 19:05:14 dist-test-slave-rn98 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363114, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.253@KRBTEST.COM
I20260925 19:05:14.872956 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:43897
--local_ip_for_outbound_sockets=127.0.196.194
--tserver_master_addrs=127.0.196.254:38655
--webserver_port=45633
--webserver_interface=127.0.196.194
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.194.keytab
--principal=oryx/127.0.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:33121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.196.254:38655,127.0.196.253:36251
--tserver_master_addrs=127.0.196.254:38655,127.0.196.253:36251,127.0.196.252:33081 with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 19:05:14 dist-test-slave-rn98 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363114, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.252@KRBTEST.COM
I20260925 19:05:14.875918 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.863688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:54891 (local address 127.0.196.253:36251)
0925 19:05:14.863753 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:14.863755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:14.863821 (+ 66us) server_negotiation.cc:407] Connection header received
0925 19:05:14.865027 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:14.865028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:14.865041 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:14.865072 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:14.867961 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.868661 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.869598 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.869892 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.869938 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.872552 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:14.872553 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:14.872554 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:14.872562 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:14.874160 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.874731 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.874733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.874733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.874795 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.875091 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.875092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.875093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.875248 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:14.875302 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:14.875702 (+ 400us) server_negotiation.cc:299] Negotiation successful
0925 19:05:14.875785 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:05:14.876368 6936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.863584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:40615 (local address 127.0.196.254:38655)
0925 19:05:14.863632 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:14.863634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:14.864103 (+ 469us) server_negotiation.cc:407] Connection header received
0925 19:05:14.865099 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:14.865100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:14.865113 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:14.865143 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:14.866312 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.867040 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.868529 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.868900 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.868972 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.871678 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:14.871679 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:14.871680 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:14.871688 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:14.875056 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.875474 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.875475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.875476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.875549 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.875746 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.875747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.875747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.875884 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:14.875941 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:14.876139 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 19:05:14.876228 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:05:14.876536 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.863310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:36251 (local address 127.0.196.193:54891)
0925 19:05:14.863707 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:14.863752 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:14.864902 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:14.865263 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:14.865267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:14.865825 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:14.866134 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.866140 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.868739 (+ 2599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.868741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.869497 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.869500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.870030 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.870031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.870091 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.871161 (+ 1070us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:14.871167 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:14.872462 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:14.874239 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.874248 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.874250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.874612 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.874873 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.874873 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.874874 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.875013 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.875336 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:14.875343 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:14.875529 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 19:05:14.876070 (+ 541us) client_negotiation.cc:240] Negotiation successful
0925 19:05:14.876253 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":35,"thread_start_us":187,"threads_started":1}
I20260925 19:05:14.876767 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.863505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.196.193:40615)
0925 19:05:14.864022 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:14.864031 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:14.864904 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:14.865266 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:14.865268 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:14.865785 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:14.866134 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.866139 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.867166 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.867167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.868431 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.868434 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.868984 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.868985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.869118 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.869752 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:14.869762 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:14.871563 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:14.875163 (+ 3600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.875164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.875164 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.875388 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.875620 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.875621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.875621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.875682 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.875987 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:14.875988 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:14.876032 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 19:05:14.876500 (+ 468us) client_negotiation.cc:240] Negotiation successful
0925 19:05:14.876634 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":43,"thread_start_us":61,"threads_started":1}
I20260925 19:05:14.877070 7863 heartbeater.cc:347] Connected to a master server at 127.0.196.253:36251
I20260925 19:05:14.877142 7862 heartbeater.cc:347] Connected to a master server at 127.0.196.254:38655
I20260925 19:05:14.877208 7862 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:14.877208 7863 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:14.877432 7863 heartbeater.cc:511] Master 127.0.196.253:36251 requested a full tablet report, sending...
I20260925 19:05:14.877725 7862 heartbeater.cc:511] Master 127.0.196.254:38655 requested a full tablet report, sending...
I20260925 19:05:14.877955 7494 ts_manager.cc:194] Registered new tserver with Master: 667b4a82651c4a26a717f37627d060d3 (127.0.196.193:34699)
I20260925 19:05:14.878286 6896 ts_manager.cc:194] Re-registered known tserver with Master: 667b4a82651c4a26a717f37627d060d3 (127.0.196.193:34699)
I20260925 19:05:14.878885 6896 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.196.193@KRBTEST.COM'} at 127.0.196.193:40615
I20260925 19:05:14.880606 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.863310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:33081 (local address 127.0.196.193:51239)
0925 19:05:14.863906 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:14.863914 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:14.864909 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:14.865265 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:14.865268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:14.865785 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:14.866134 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.866139 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.872730 (+ 6591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.872732 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.873619 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:14.873621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.874235 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.874236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:14.874303 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.874750 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:14.874755 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:14.876801 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:14.879494 (+ 2693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.879495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.879495 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.879715 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.879946 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:14.879947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:14.879947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:14.880016 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:14.880346 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:14.880347 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:14.880377 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 19:05:14.880435 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 19:05:14.880520 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":54,"thread_start_us":164,"threads_started":1}
I20260925 19:05:14.880709 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.863673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:51239 (local address 127.0.196.252:33081)
0925 19:05:14.863742 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:14.863745 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:14.864057 (+ 312us) server_negotiation.cc:407] Connection header received
0925 19:05:14.865031 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:14.865032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:14.865049 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:14.865082 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:14.866223 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.867071 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.873726 (+ 6655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:14.874095 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:14.874174 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:14.877160 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:14.877162 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:14.877162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:14.877172 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:14.879384 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.879809 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.879811 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.879812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.879872 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:14.880091 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:14.880092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:14.880093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:14.880257 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:14.880346 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:14.880467 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 19:05:14.880581 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 19:05:14.881256 7861 heartbeater.cc:347] Connected to a master server at 127.0.196.252:33081
I20260925 19:05:14.881322 7861 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:14.881393 7861 heartbeater.cc:511] Master 127.0.196.252:33081 requested a full tablet report, sending...
I20260925 19:05:14.881803 7680 ts_manager.cc:194] Registered new tserver with Master: 667b4a82651c4a26a717f37627d060d3 (127.0.196.193:34699)
W20260925 19:05:14.959182 7868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:14.959405 7868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:14.959431 7868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:14.960618 7868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:14.960664 7868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:14.960690 7868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:05:14.962303 7868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:33121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.194.keytab
--principal=oryx/127.0.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:43897
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=45633
--tserver_master_addrs=127.0.196.254:38655,127.0.196.253:36251,127.0.196.252:33081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:14.962507 7868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:14.962800 7868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:14.968053 7876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:14.968053 7873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:14.968261 7868 server_base.cc:1034] running on GCE node
W20260925 19:05:14.968063 7874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:14.968534 7868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:14.968813 7868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:14.969959 7868 hybrid_clock.cc:648] HybridClock initialized: now 1790363114969945 us; error 34 us; skew 500 ppm
Sep 25 19:05:14 dist-test-slave-rn98 krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363114, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:14.972541 7868 init.cc:377] Logged in from keytab as oryx/127.0.196.194@KRBTEST.COM (short username oryx)
I20260925 19:05:14.972998 7868 webserver.cc:460] Webserver started at http://127.0.196.194:45633/ using document root <none> and password file <none>
I20260925 19:05:14.973239 7868 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:14.973336 7868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:14.974828 7868 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:05:14.975484 7883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:14.975651 7868 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:14.975770 7868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "6a29d1c1c84644fea8558af4f2c9823f"
format_stamp: "Formatted at 2026-09-25 19:05:13 on dist-test-slave-rn98"
server_key: "1a89b8971c3c2bb3e0bab023ea45de19"
server_key_iv: "b23db239671011dac7de370a03ec7222"
server_key_version: "encryptionkey@0"
I20260925 19:05:14.976169 7868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:14.987241 7868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:14.989470 7868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:14.989640 7868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:14.989886 7868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:14.990247 7868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:14.990304 7868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:14.990350 7868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:14.990373 7868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:14.996574 7868 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:43897
I20260925 19:05:14.996655 7996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:43897 every 8 connection(s)
I20260925 19:05:14.997027 7868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
I20260925 19:05:14.998505 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 7868
I20260925 19:05:14.998708 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 7226
Sep 25 19:05:14 dist-test-slave-rn98 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363114, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.252@KRBTEST.COM
Sep 25 19:05:14 dist-test-slave-rn98 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363114, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM
Sep 25 19:05:15 dist-test-slave-rn98 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363114, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.253@KRBTEST.COM
I20260925 19:05:15.008200 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:36681
--local_ip_for_outbound_sockets=127.0.196.195
--tserver_master_addrs=127.0.196.254:38655
--webserver_port=40219
--webserver_interface=127.0.196.195
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.195.keytab
--principal=oryx/127.0.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:33121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.196.254:38655,127.0.196.253:36251
--tserver_master_addrs=127.0.196.254:38655,127.0.196.253:36251,127.0.196.252:33081 with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:05:15.010486 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.997799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:46045 (local address 127.0.196.252:33081)
0925 19:05:14.997847 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:14.997849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:14.998174 (+ 325us) server_negotiation.cc:407] Connection header received
0925 19:05:15.000484 (+ 2310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:15.000485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:15.000500 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:15.000532 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:15.001080 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.001877 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.003096 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.003471 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.003516 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.006606 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:15.006608 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:15.006608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:15.006643 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:15.008532 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:15.009348 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:15.009350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:15.009350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:15.009411 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:15.009729 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:15.009730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:15.009730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:15.009861 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:15.009938 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:15.010267 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 19:05:15.010372 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:05:15.010800 6936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.997798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:47067 (local address 127.0.196.254:38655)
0925 19:05:14.997847 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:14.997849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:14.998427 (+ 578us) server_negotiation.cc:407] Connection header received
0925 19:05:15.000466 (+ 2039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:15.000468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:15.000481 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:15.000538 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:15.001038 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.001758 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.002853 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.003192 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.003230 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.007132 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:15.007134 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:15.007134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:15.007144 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:15.009528 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:15.009927 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:15.009928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:15.009929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:15.009987 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:15.010212 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:15.010213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:15.010213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:15.010352 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:15.010414 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:15.010598 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 19:05:15.010712 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:05:15.011181 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.997834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.196.194:47067)
0925 19:05:14.998345 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:14.998375 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:15.000385 (+ 2010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:15.000590 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:15.000591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:15.000864 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:15.000936 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:15.000938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.001858 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.001859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:15.002742 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:15.002745 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.003323 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.003324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:15.003498 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.004221 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:15.004233 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:15.006941 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:15.009623 (+ 2682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:15.009624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:15.009625 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:15.009841 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:15.010070 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:15.010070 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:15.010071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:15.010126 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:15.010442 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:15.010442 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:15.010478 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 19:05:15.010876 (+ 398us) client_negotiation.cc:240] Negotiation successful
0925 19:05:15.010971 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":32,"thread_start_us":92,"threads_started":1}
I20260925 19:05:15.011247 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.998243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:39615 (local address 127.0.196.253:36251)
0925 19:05:14.998287 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:14.998289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:14.999346 (+ 1057us) server_negotiation.cc:407] Connection header received
0925 19:05:14.999408 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:14.999409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:14.999419 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:14.999461 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:15.000373 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.001162 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.002853 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.003173 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.003223 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.007406 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:15.007407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:15.007408 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:15.007418 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:15.009908 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:15.010278 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:15.010279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:15.010280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:15.010338 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:15.010544 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:15.010545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:15.010545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:15.010721 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:15.010785 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:15.011037 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 19:05:15.011124 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:05:15.011592 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.998166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:36251 (local address 127.0.196.194:39615)
0925 19:05:14.999278 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:14.999289 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:14.999347 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:14.999544 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:14.999549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:14.999980 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:15.000284 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:15.000289 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.001298 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.001299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:15.002742 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:15.002745 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.003344 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.003344 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:15.003498 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.004221 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:15.004233 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:15.007296 (+ 3063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:15.009990 (+ 2694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:15.009990 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:15.009991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:15.010198 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:15.010409 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:15.010409 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:15.010409 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:15.010461 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:15.010823 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:15.010824 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:15.010919 (+ 95us) client_negotiation.cc:769] Sending connection context
0925 19:05:15.011354 (+ 435us) client_negotiation.cc:240] Negotiation successful
0925 19:05:15.011470 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1097,"mutex_wait_us":20,"thread_start_us":79,"threads_started":1}
I20260925 19:05:15.012059 7999 heartbeater.cc:347] Connected to a master server at 127.0.196.253:36251
I20260925 19:05:15.012161 7999 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:15.012313 7999 heartbeater.cc:511] Master 127.0.196.253:36251 requested a full tablet report, sending...
I20260925 19:05:15.012364 7998 heartbeater.cc:347] Connected to a master server at 127.0.196.254:38655
I20260925 19:05:15.012413 7998 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:15.012472 7998 heartbeater.cc:511] Master 127.0.196.254:38655 requested a full tablet report, sending...
I20260925 19:05:15.012701 6896 ts_manager.cc:194] Re-registered known tserver with Master: 6a29d1c1c84644fea8558af4f2c9823f (127.0.196.194:43897)
I20260925 19:05:15.012779 7494 ts_manager.cc:194] Registered new tserver with Master: 6a29d1c1c84644fea8558af4f2c9823f (127.0.196.194:43897)
I20260925 19:05:15.013028 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:14.997835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:33081 (local address 127.0.196.194:46045)
0925 19:05:14.998036 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:14.998073 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:14.998934 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:15.000599 (+ 1665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:15.000600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:15.000902 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:15.000972 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:15.000974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.001976 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.001977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:15.002983 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:15.002985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.003629 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.003630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:15.003723 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.004357 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:15.004362 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:15.006482 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:15.008880 (+ 2398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:15.008889 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:15.008892 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:15.009254 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:15.009484 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:15.009485 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:15.009485 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:15.009644 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:15.009940 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:15.009949 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:15.010113 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 19:05:15.010241 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 19:05:15.010433 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"spinlock_wait_cycles":8064,"thread_start_us":97,"threads_started":1}
I20260925 19:05:15.013223 6896 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.196.194@KRBTEST.COM'} at 127.0.196.194:47067
I20260925 19:05:15.013397 7997 heartbeater.cc:347] Connected to a master server at 127.0.196.252:33081
I20260925 19:05:15.013453 7997 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:15.013533 7997 heartbeater.cc:511] Master 127.0.196.252:33081 requested a full tablet report, sending...
I20260925 19:05:15.013811 7680 ts_manager.cc:194] Registered new tserver with Master: 6a29d1c1c84644fea8558af4f2c9823f (127.0.196.194:43897)
I20260925 19:05:15.085071 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:15.081909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:33081 (local address 127.0.0.1:59326)
0925 19:05:15.081992 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:15.082007 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:15.082109 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:15.082328 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:15.082329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:15.082331 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:15.082391 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:15.082393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.083301 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.083302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:15.084104 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:15.084106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.084789 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.084790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:15.084858 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.084862 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:05:15.084929 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 19:05:15.084985 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 19:05:15.085071 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:15.082026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59326 (local address 127.0.196.252:33081)
0925 19:05:15.082095 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:15.082098 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:15.082111 (+ 13us) server_negotiation.cc:407] Connection header received
0925 19:05:15.082188 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:15.082189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:15.082217 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:15.082298 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:15.082500 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.083191 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.084211 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.084691 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.084787 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.084865 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:15.084936 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 19:05:15.084985 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 19:05:15.085517 7690 raft_consensus.cc:1203] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 19:05:15.085685 7719 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c9e241c502f4fed8aae1e80224e05b9. Latest consensus state: current_term: 1 leader_uuid: "2c9e241c502f4fed8aae1e80224e05b9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: NON_VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: true } } }
I20260925 19:05:15.085815 7719 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [sys.catalog]: This master's current role is: LEARNER
W20260925 19:05:15.096493 8004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:15.096726 8004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:15.096752 8004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:15.098116 8004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:15.098165 8004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:15.098196 8004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:05:15.100028 8004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:33121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/krb5kdc/oryx_127.0.196.195.keytab
--principal=oryx/127.0.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:36681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=40219
--tserver_master_addrs=127.0.196.254:38655,127.0.196.253:36251,127.0.196.252:33081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:15.100279 8004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:15.100549 8004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:15.105894 8010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:15.105851 8009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:15.105878 8012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:15.106316 8004 server_base.cc:1034] running on GCE node
I20260925 19:05:15.106528 8004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:15.106819 8004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:15.107977 8004 hybrid_clock.cc:648] HybridClock initialized: now 1790363115107960 us; error 32 us; skew 500 ppm
Sep 25 19:05:15 dist-test-slave-rn98 krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363115, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:15.110787 8004 init.cc:377] Logged in from keytab as oryx/127.0.196.195@KRBTEST.COM (short username oryx)
I20260925 19:05:15.111305 8004 webserver.cc:460] Webserver started at http://127.0.196.195:40219/ using document root <none> and password file <none>
I20260925 19:05:15.111550 8004 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:15.111634 8004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:15.113143 8004 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:15.113806 8019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:15.113992 8004 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:15.114085 8004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "4e5e4690e8104ca09a7592a858d2983c"
format_stamp: "Formatted at 2026-09-25 19:05:13 on dist-test-slave-rn98"
server_key: "9c1bcd94e08beffeca669bd646b07076"
server_key_iv: "5cb1dde3dec543c200ba33e8ce9562fa"
server_key_version: "encryptionkey@0"
I20260925 19:05:15.114428 8004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:15.136121 8004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:15.138410 8004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:15.138568 8004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:15.138888 8004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:15.139307 8004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:15.139369 8004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:15.139418 8004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:15.139456 8004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:15.146917 8004 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:36681
I20260925 19:05:15.146997 8132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:36681 every 8 connection(s)
I20260925 19:05:15.147377 8004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
Sep 25 19:05:15 dist-test-slave-rn98 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363115, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM
Sep 25 19:05:15 dist-test-slave-rn98 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363115, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.252@KRBTEST.COM
Sep 25 19:05:15 dist-test-slave-rn98 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363115, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.253@KRBTEST.COM
I20260925 19:05:15.157117 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 8004
I20260925 19:05:15.159075 6936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:15.148249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:34369 (local address 127.0.196.254:38655)
0925 19:05:15.148310 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:15.148313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:15.149452 (+ 1139us) server_negotiation.cc:407] Connection header received
0925 19:05:15.149482 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:15.149484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:15.149497 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:15.149578 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:15.150292 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.151081 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.152661 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.152863 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.152950 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.155449 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:15.155451 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:15.155451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:15.155461 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:15.157221 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:15.157765 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:15.157766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:15.157767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:15.157828 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:15.158129 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:15.158130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:15.158130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:15.158285 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:15.158342 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:15.158887 (+ 545us) server_negotiation.cc:299] Negotiation successful
0925 19:05:15.158971 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 19:05:15.159328 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:15.148182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.196.195:34369)
0925 19:05:15.149342 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:15.149352 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:15.149404 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:15.149618 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:15.149620 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:15.149885 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:15.150171 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:15.150179 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.151184 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.151185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:15.152111 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:15.152113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.152951 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.152952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:15.153028 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.153643 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:15.153649 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:15.155344 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:15.157316 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:15.157323 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:15.157326 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:15.157670 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:15.157898 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:15.157899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:15.157899 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:15.158046 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:15.158394 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:15.158401 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:15.158691 (+ 290us) client_negotiation.cc:769] Sending connection context
0925 19:05:15.158862 (+ 171us) client_negotiation.cc:240] Negotiation successful
0925 19:05:15.159045 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1128,"mutex_wait_us":31,"thread_start_us":107,"threads_started":1}
I20260925 19:05:15.159777 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:15.148539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:45943 (local address 127.0.196.252:33081)
0925 19:05:15.148608 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:15.148610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:15.148716 (+ 106us) server_negotiation.cc:407] Connection header received
0925 19:05:15.149260 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:15.149261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:15.149273 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:15.149334 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:15.150273 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.150924 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.152099 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.152439 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.152525 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.155828 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:15.155829 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:15.155829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:15.155838 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:15.157687 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:15.158169 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:15.158170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:15.158171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:15.158233 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:15.158441 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:15.158442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:15.158443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:15.158586 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:15.158715 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:15.158947 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 19:05:15.159043 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 19:05:15.159907 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:15.148600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:58189 (local address 127.0.196.253:36251)
0925 19:05:15.148664 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:15.148666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:15.149853 (+ 1187us) server_negotiation.cc:407] Connection header received
0925 19:05:15.149889 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:15.149890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:15.149902 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:15.149969 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:15.150291 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.151102 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.152197 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.152532 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.152610 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.156621 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:15.156623 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:15.156623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:15.156632 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:15.158413 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:15.158925 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:15.158927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:15.158939 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:15.159014 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:15.159258 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:15.159258 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:15.159259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:15.159406 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:15.159461 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:15.159700 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 19:05:15.159799 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 19:05:15.159912 8134 heartbeater.cc:347] Connected to a master server at 127.0.196.254:38655
I20260925 19:05:15.160059 8134 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:15.160233 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:15.148181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:33081 (local address 127.0.196.195:45943)
0925 19:05:15.148578 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:15.148626 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:15.149143 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:15.149474 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:15.149480 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:15.149886 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:15.150170 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:15.150176 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.151048 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.151049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:15.151984 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:15.151986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.152540 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.152540 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:15.152722 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.153476 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:15.153483 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:15.155709 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:15.157842 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:15.157843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:15.157844 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:15.158067 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:15.158317 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:15.158318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:15.158318 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:15.158380 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:15.158733 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:15.158734 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:15.158786 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 19:05:15.159993 (+ 1207us) client_negotiation.cc:240] Negotiation successful
0925 19:05:15.160120 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":1,"thread_start_us":116,"threads_started":1}
I20260925 19:05:15.160275 8134 heartbeater.cc:511] Master 127.0.196.254:38655 requested a full tablet report, sending...
I20260925 19:05:15.160624 8139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:15.148182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:36251 (local address 127.0.196.195:58189)
0925 19:05:15.149764 (+ 1582us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:15.149774 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:15.149837 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:15.149986 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:15.149986 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:15.150143 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:15.150193 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:15.150195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.151184 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.151185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:15.152111 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:15.152113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.152608 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.152608 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:15.152722 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.153469 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:15.153481 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:15.156500 (+ 3019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:15.158500 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:15.158500 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:15.158501 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:15.158833 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:15.159102 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:15.159103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:15.159104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:15.159167 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:15.159524 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:15.159525 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:15.159567 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 19:05:15.160030 (+ 463us) client_negotiation.cc:240] Negotiation successful
0925 19:05:15.160487 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1265,"mutex_wait_us":349,"thread_start_us":83,"threads_started":1}
I20260925 19:05:15.160689 6896 ts_manager.cc:194] Re-registered known tserver with Master: 4e5e4690e8104ca09a7592a858d2983c (127.0.196.195:36681)
I20260925 19:05:15.161059 8133 heartbeater.cc:347] Connected to a master server at 127.0.196.252:33081
I20260925 19:05:15.161103 8133 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:15.161164 8133 heartbeater.cc:511] Master 127.0.196.252:33081 requested a full tablet report, sending...
I20260925 19:05:15.161248 6896 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.196.195@KRBTEST.COM'} at 127.0.196.195:34369
I20260925 19:05:15.161481 7680 ts_manager.cc:194] Registered new tserver with Master: 4e5e4690e8104ca09a7592a858d2983c (127.0.196.195:36681)
I20260925 19:05:15.161566 8135 heartbeater.cc:347] Connected to a master server at 127.0.196.253:36251
I20260925 19:05:15.161607 8135 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:15.161679 8135 heartbeater.cc:511] Master 127.0.196.253:36251 requested a full tablet report, sending...
I20260925 19:05:15.161955 7494 ts_manager.cc:194] Registered new tserver with Master: 4e5e4690e8104ca09a7592a858d2983c (127.0.196.195:36681)
I20260925 19:05:15.169569 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:15.160668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59340 (local address 127.0.196.252:33081)
0925 19:05:15.160703 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:15.160706 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:15.160881 (+ 175us) server_negotiation.cc:407] Connection header received
0925 19:05:15.160982 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:15.160983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:15.160994 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:15.161042 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:15.162013 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.162889 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.164008 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.164320 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.164362 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.166213 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:15.166214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:15.166215 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:15.166222 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:15.168005 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:15.168670 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:15.168670 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:15.168671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:15.168720 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:15.168934 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:15.168935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:15.168936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:15.169111 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:15.169242 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:15.169354 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 19:05:15.169446 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:05:15.169734 6936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:15.159906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42206 (local address 127.0.196.254:38655)
0925 19:05:15.159947 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:15.159949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:15.161533 (+ 1584us) server_negotiation.cc:407] Connection header received
0925 19:05:15.161748 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:15.161749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:15.161759 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:15.161790 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:15.162659 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.163365 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.164367 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.164672 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.164711 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.166282 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:15.166283 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:15.166284 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:15.166294 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:15.168402 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:15.168877 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:15.168879 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:15.168879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:15.168933 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:15.169113 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:15.169114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:15.169114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:15.169255 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:15.169346 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:15.169531 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 19:05:15.169636 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:05:15.171588 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:15.161133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41988 (local address 127.0.196.253:36251)
0925 19:05:15.161171 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:15.161173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:15.161640 (+ 467us) server_negotiation.cc:407] Connection header received
0925 19:05:15.162257 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:15.162258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:15.162268 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:15.162323 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:15.163406 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.164061 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.165219 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.165534 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.165586 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.167487 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:15.167488 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:15.167489 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:15.167497 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:15.169410 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:15.170657 (+ 1247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:15.170659 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:15.170659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:15.170725 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:15.170963 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:15.170963 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:15.170964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:15.171132 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:15.171239 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:15.171406 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 19:05:15.171489 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:05:15.171975 6896 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:05:15.173528 6896 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:05:15.174235 7642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c9e241c502f4fed8aae1e80224e05b9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: NON_VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: true } } }
I20260925 19:05:15.174330 7642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:15.175654 7642 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9: attempting to promote NON_VOTER dfdd31dd8df548429c1f60d15fc974fa to VOTER
I20260925 19:05:15.175788 7642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: false } }
I20260925 19:05:15.176230 7690 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [term 1 LEARNER]: Refusing update from remote peer 2c9e241c502f4fed8aae1e80224e05b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 19:05:15.176427 8150 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 19:05:15.176836 7504 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 1 FOLLOWER]: Refusing update from remote peer 2c9e241c502f4fed8aae1e80224e05b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 19:05:15.176985 8150 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, dfdd31dd8df548429c1f60d15fc974fa (127.0.196.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: false } } }
I20260925 19:05:15.177120 7642 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 19:05:15.177147 7690 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, dfdd31dd8df548429c1f60d15fc974fa (127.0.196.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: false } } }
I20260925 19:05:15.177459 7504 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, dfdd31dd8df548429c1f60d15fc974fa (127.0.196.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: false } } }
I20260925 19:05:15.178103 8150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2c9e241c502f4fed8aae1e80224e05b9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: false } } }
I20260925 19:05:15.178191 8150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:15.178248 8150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c9e241c502f4fed8aae1e80224e05b9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: false } } }
I20260925 19:05:15.178254 7719 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c9e241c502f4fed8aae1e80224e05b9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: false } } }
I20260925 19:05:15.178328 8150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:15.178337 7719 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:05:15.178498 8158 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c9e241c502f4fed8aae1e80224e05b9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: false } } }
I20260925 19:05:15.178601 8158 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:05:15.178834 8149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c9e241c502f4fed8aae1e80224e05b9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: false } } }
I20260925 19:05:15.178920 8149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:15.184041 8162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:15.180026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59904 (local address 127.0.196.194:43897)
0925 19:05:15.180442 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:15.180446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:15.180485 (+ 39us) server_negotiation.cc:407] Connection header received
0925 19:05:15.180585 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:15.180587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:15.180599 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:15.180688 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:15.180827 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.181518 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.183172 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.183682 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.183780 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.183867 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:15.183918 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 19:05:15.183958 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":75,"threads_started":1}
I20260925 19:05:15.184031 8159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:15.179813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:43897 (local address 127.0.0.1:59904)
0925 19:05:15.180429 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:15.180440 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:15.180499 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:15.180700 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:15.180701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:15.180703 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:15.180749 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:15.180751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.181632 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.181633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:15.183079 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:15.183081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.183783 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.183784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:15.183840 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.183845 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:05:15.183902 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 19:05:15.183945 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":35,"thread_start_us":92,"threads_started":1}
I20260925 19:05:15.184170 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:15.179732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:34699 (local address 127.0.0.1:32926)
0925 19:05:15.179779 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:15.179791 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:15.179833 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:15.180199 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:15.180200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:15.180202 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:15.180258 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:15.180260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.181700 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.181700 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:15.182892 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:15.182895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.183939 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.183940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:15.184008 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.184012 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:05:15.184047 (+ 35us) client_negotiation.cc:240] Negotiation successful
0925 19:05:15.184097 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":31}
I20260925 19:05:15.184453 8160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:15.179791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:36681 (local address 127.0.0.1:45332)
0925 19:05:15.179946 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:15.179957 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:15.180000 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:15.180789 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:15.180789 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:15.180791 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:15.180824 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:15.180825 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.182026 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.182028 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:15.183144 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:15.183147 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.184225 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.184226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:15.184286 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.184290 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:05:15.184345 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 19:05:15.184388 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":2,"thread_start_us":98,"threads_started":1}
I20260925 19:05:15.184525 8161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:15.179840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32926 (local address 127.0.196.193:34699)
0925 19:05:15.180016 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:15.180021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:15.180031 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:05:15.180077 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:15.180080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:15.180097 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:15.180194 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:15.180354 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.181608 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.182974 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.183826 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.184257 (+ 431us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.184346 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:15.184375 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 19:05:15.184420 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":69,"threads_started":1}
I20260925 19:05:15.184754 8163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:15.180029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45332 (local address 127.0.196.195:36681)
0925 19:05:15.180626 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:15.180632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:15.180643 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:05:15.180684 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:15.180686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:15.180702 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:15.180778 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:15.180877 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.181929 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.183242 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.184046 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.184110 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.184188 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:15.184638 (+ 450us) server_negotiation.cc:299] Negotiation successful
0925 19:05:15.184692 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":84,"threads_started":1}
I20260925 19:05:15.185149 7931 tablet_service.cc:1467] Processing CreateTablet for tablet 62a8114d15154ba298d6cee5bb4c7097 (DEFAULT_TABLE table=test-table [id=1554b82b7b624ca3a0a28768cdc45166]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:05:15.185185 7794 tablet_service.cc:1467] Processing CreateTablet for tablet 62a8114d15154ba298d6cee5bb4c7097 (DEFAULT_TABLE table=test-table [id=1554b82b7b624ca3a0a28768cdc45166]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:05:15.185536 7794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a8114d15154ba298d6cee5bb4c7097. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:15.185539 7931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a8114d15154ba298d6cee5bb4c7097. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:15.187345 8067 tablet_service.cc:1467] Processing CreateTablet for tablet 62a8114d15154ba298d6cee5bb4c7097 (DEFAULT_TABLE table=test-table [id=1554b82b7b624ca3a0a28768cdc45166]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:05:15.187704 8067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a8114d15154ba298d6cee5bb4c7097. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:15.188654 8164 tablet_bootstrap.cc:492] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3: Bootstrap starting.
I20260925 19:05:15.189030 8165 tablet_bootstrap.cc:492] T 62a8114d15154ba298d6cee5bb4c7097 P 6a29d1c1c84644fea8558af4f2c9823f: Bootstrap starting.
I20260925 19:05:15.189595 8164 tablet_bootstrap.cc:654] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:15.189718 8165 tablet_bootstrap.cc:654] T 62a8114d15154ba298d6cee5bb4c7097 P 6a29d1c1c84644fea8558af4f2c9823f: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:15.189960 8164 log.cc:824] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:15.190044 8165 log.cc:824] T 62a8114d15154ba298d6cee5bb4c7097 P 6a29d1c1c84644fea8558af4f2c9823f: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:15.190863 8165 tablet_bootstrap.cc:492] T 62a8114d15154ba298d6cee5bb4c7097 P 6a29d1c1c84644fea8558af4f2c9823f: No bootstrap required, opened a new log
I20260925 19:05:15.190871 8164 tablet_bootstrap.cc:492] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3: No bootstrap required, opened a new log
I20260925 19:05:15.190935 8164 ts_tablet_manager.cc:1397] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:05:15.190944 8165 ts_tablet_manager.cc:1397] T 62a8114d15154ba298d6cee5bb4c7097 P 6a29d1c1c84644fea8558af4f2c9823f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:05:15.191061 8168 tablet_bootstrap.cc:492] T 62a8114d15154ba298d6cee5bb4c7097 P 4e5e4690e8104ca09a7592a858d2983c: Bootstrap starting.
I20260925 19:05:15.192066 8168 tablet_bootstrap.cc:654] T 62a8114d15154ba298d6cee5bb4c7097 P 4e5e4690e8104ca09a7592a858d2983c: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:15.192405 8168 log.cc:824] T 62a8114d15154ba298d6cee5bb4c7097 P 4e5e4690e8104ca09a7592a858d2983c: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:15.193279 8168 tablet_bootstrap.cc:492] T 62a8114d15154ba298d6cee5bb4c7097 P 4e5e4690e8104ca09a7592a858d2983c: No bootstrap required, opened a new log
I20260925 19:05:15.193356 8168 ts_tablet_manager.cc:1397] T 62a8114d15154ba298d6cee5bb4c7097 P 4e5e4690e8104ca09a7592a858d2983c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:05:15.193326 8165 raft_consensus.cc:348] T 62a8114d15154ba298d6cee5bb4c7097 P 6a29d1c1c84644fea8558af4f2c9823f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b4a82651c4a26a717f37627d060d3" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34699 } } peers { permanent_uuid: "4e5e4690e8104ca09a7592a858d2983c" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 36681 } } peers { permanent_uuid: "6a29d1c1c84644fea8558af4f2c9823f" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43897 } }
I20260925 19:05:15.193517 8165 raft_consensus.cc:374] T 62a8114d15154ba298d6cee5bb4c7097 P 6a29d1c1c84644fea8558af4f2c9823f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:15.193558 8165 raft_consensus.cc:729] T 62a8114d15154ba298d6cee5bb4c7097 P 6a29d1c1c84644fea8558af4f2c9823f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a29d1c1c84644fea8558af4f2c9823f, State: Initialized, Role: FOLLOWER
I20260925 19:05:15.193583 8164 raft_consensus.cc:348] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b4a82651c4a26a717f37627d060d3" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34699 } } peers { permanent_uuid: "4e5e4690e8104ca09a7592a858d2983c" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 36681 } } peers { permanent_uuid: "6a29d1c1c84644fea8558af4f2c9823f" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43897 } }
I20260925 19:05:15.193758 8164 raft_consensus.cc:374] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:15.193804 8164 raft_consensus.cc:729] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 667b4a82651c4a26a717f37627d060d3, State: Initialized, Role: FOLLOWER
I20260925 19:05:15.193912 8164 consensus_queue.cc:260] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b4a82651c4a26a717f37627d060d3" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34699 } } peers { permanent_uuid: "4e5e4690e8104ca09a7592a858d2983c" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 36681 } } peers { permanent_uuid: "6a29d1c1c84644fea8558af4f2c9823f" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43897 } }
I20260925 19:05:15.193909 8165 consensus_queue.cc:260] T 62a8114d15154ba298d6cee5bb4c7097 P 6a29d1c1c84644fea8558af4f2c9823f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b4a82651c4a26a717f37627d060d3" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34699 } } peers { permanent_uuid: "4e5e4690e8104ca09a7592a858d2983c" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 36681 } } peers { permanent_uuid: "6a29d1c1c84644fea8558af4f2c9823f" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43897 } }
I20260925 19:05:15.194207 8164 ts_tablet_manager.cc:1428] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 19:05:15.194226 8165 ts_tablet_manager.cc:1428] T 62a8114d15154ba298d6cee5bb4c7097 P 6a29d1c1c84644fea8558af4f2c9823f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 19:05:15.194450 7862 heartbeater.cc:503] Master 127.0.196.254:38655 was elected leader, sending a full tablet report...
I20260925 19:05:15.194802 7998 heartbeater.cc:503] Master 127.0.196.254:38655 was elected leader, sending a full tablet report...
I20260925 19:05:15.196643 8168 raft_consensus.cc:348] T 62a8114d15154ba298d6cee5bb4c7097 P 4e5e4690e8104ca09a7592a858d2983c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b4a82651c4a26a717f37627d060d3" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34699 } } peers { permanent_uuid: "4e5e4690e8104ca09a7592a858d2983c" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 36681 } } peers { permanent_uuid: "6a29d1c1c84644fea8558af4f2c9823f" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43897 } }
I20260925 19:05:15.196853 8168 raft_consensus.cc:374] T 62a8114d15154ba298d6cee5bb4c7097 P 4e5e4690e8104ca09a7592a858d2983c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:15.196909 8168 raft_consensus.cc:729] T 62a8114d15154ba298d6cee5bb4c7097 P 4e5e4690e8104ca09a7592a858d2983c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e5e4690e8104ca09a7592a858d2983c, State: Initialized, Role: FOLLOWER
I20260925 19:05:15.197068 8168 consensus_queue.cc:260] T 62a8114d15154ba298d6cee5bb4c7097 P 4e5e4690e8104ca09a7592a858d2983c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b4a82651c4a26a717f37627d060d3" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34699 } } peers { permanent_uuid: "4e5e4690e8104ca09a7592a858d2983c" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 36681 } } peers { permanent_uuid: "6a29d1c1c84644fea8558af4f2c9823f" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43897 } }
I20260925 19:05:15.197361 8168 ts_tablet_manager.cc:1428] T 62a8114d15154ba298d6cee5bb4c7097 P 4e5e4690e8104ca09a7592a858d2983c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260925 19:05:15.197391 8134 heartbeater.cc:503] Master 127.0.196.254:38655 was elected leader, sending a full tablet report...
I20260925 19:05:15.208506 8170 raft_consensus.cc:482] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:05:15.208629 8170 raft_consensus.cc:504] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b4a82651c4a26a717f37627d060d3" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34699 } } peers { permanent_uuid: "4e5e4690e8104ca09a7592a858d2983c" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 36681 } } peers { permanent_uuid: "6a29d1c1c84644fea8558af4f2c9823f" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43897 } }
I20260925 19:05:15.209012 8170 leader_election.cc:288] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e5e4690e8104ca09a7592a858d2983c (127.0.196.195:36681), 6a29d1c1c84644fea8558af4f2c9823f (127.0.196.194:43897)
I20260925 19:05:15.211678 8172 raft_consensus.cc:482] T 62a8114d15154ba298d6cee5bb4c7097 P 4e5e4690e8104ca09a7592a858d2983c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:05:15.211817 8172 raft_consensus.cc:504] T 62a8114d15154ba298d6cee5bb4c7097 P 4e5e4690e8104ca09a7592a858d2983c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b4a82651c4a26a717f37627d060d3" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34699 } } peers { permanent_uuid: "4e5e4690e8104ca09a7592a858d2983c" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 36681 } } peers { permanent_uuid: "6a29d1c1c84644fea8558af4f2c9823f" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43897 } }
I20260925 19:05:15.211949 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:15.209250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:36681 (local address 127.0.196.193:45751)
0925 19:05:15.209302 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:15.209316 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:15.209367 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:15.209544 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:15.209545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:15.209547 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:15.209625 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:15.209627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.210263 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.210264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:15.211088 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:15.211091 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.211660 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.211661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:15.211755 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.211760 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:05:15.211818 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 19:05:15.211862 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 19:05:15.212297 8163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:15.209375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:45751 (local address 127.0.196.195:36681)
0925 19:05:15.209416 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:15.209418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:15.209425 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:05:15.209450 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:15.209451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:15.209459 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:15.209539 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:15.209700 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.210177 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.211183 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.211566 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.211661 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.211672 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:15.212162 (+ 490us) server_negotiation.cc:299] Negotiation successful
0925 19:05:15.212207 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:05:15.212601 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:15.209250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:43897 (local address 127.0.196.193:55105)
0925 19:05:15.209487 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:15.209497 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:15.209581 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:15.209734 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:15.209735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:15.209736 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:15.209808 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:15.209810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.210557 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.210558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:15.211429 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:15.211431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.212335 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.212336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:15.212429 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.212433 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:05:15.212485 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 19:05:15.212533 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":175}
I20260925 19:05:15.212713 8172 leader_election.cc:288] T 62a8114d15154ba298d6cee5bb4c7097 P 4e5e4690e8104ca09a7592a858d2983c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 667b4a82651c4a26a717f37627d060d3 (127.0.196.193:34699), 6a29d1c1c84644fea8558af4f2c9823f (127.0.196.194:43897)
I20260925 19:05:15.212963 8087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a8114d15154ba298d6cee5bb4c7097" candidate_uuid: "667b4a82651c4a26a717f37627d060d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e5e4690e8104ca09a7592a858d2983c" is_pre_election: true
I20260925 19:05:15.213104 8087 raft_consensus.cc:2454] T 62a8114d15154ba298d6cee5bb4c7097 P 4e5e4690e8104ca09a7592a858d2983c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 667b4a82651c4a26a717f37627d060d3 in term 0.
I20260925 19:05:15.213421 8162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:15.209311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:55105 (local address 127.0.196.194:43897)
0925 19:05:15.209362 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:15.209365 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:15.209573 (+ 208us) server_negotiation.cc:407] Connection header received
0925 19:05:15.209641 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:15.209643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:15.209653 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:15.209733 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:15.209886 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.210454 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.211593 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.212222 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.212292 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.212300 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:15.213294 (+ 994us) server_negotiation.cc:299] Negotiation successful
0925 19:05:15.213351 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:05:15.213438 7748 leader_election.cc:302] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e5e4690e8104ca09a7592a858d2983c, 667b4a82651c4a26a717f37627d060d3; no voters:
I20260925 19:05:15.213613 8170 raft_consensus.cc:2781] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:05:15.213660 8170 raft_consensus.cc:482] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:05:15.213682 8170 raft_consensus.cc:3037] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:15.213702 7951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a8114d15154ba298d6cee5bb4c7097" candidate_uuid: "667b4a82651c4a26a717f37627d060d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a29d1c1c84644fea8558af4f2c9823f" is_pre_election: true
I20260925 19:05:15.213852 7951 raft_consensus.cc:2454] T 62a8114d15154ba298d6cee5bb4c7097 P 6a29d1c1c84644fea8558af4f2c9823f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 667b4a82651c4a26a717f37627d060d3 in term 0.
I20260925 19:05:15.215564 8170 raft_consensus.cc:504] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b4a82651c4a26a717f37627d060d3" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34699 } } peers { permanent_uuid: "4e5e4690e8104ca09a7592a858d2983c" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 36681 } } peers { permanent_uuid: "6a29d1c1c84644fea8558af4f2c9823f" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43897 } }
I20260925 19:05:15.215837 8087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a8114d15154ba298d6cee5bb4c7097" candidate_uuid: "667b4a82651c4a26a717f37627d060d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e5e4690e8104ca09a7592a858d2983c"
I20260925 19:05:15.215955 8087 raft_consensus.cc:3037] T 62a8114d15154ba298d6cee5bb4c7097 P 4e5e4690e8104ca09a7592a858d2983c [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:15.216099 8161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:15.212635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:50905 (local address 127.0.196.193:34699)
0925 19:05:15.212678 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:15.212680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:15.212802 (+ 122us) server_negotiation.cc:407] Connection header received
0925 19:05:15.212836 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:15.212836 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:15.212846 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:15.212905 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:15.213099 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.213842 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.214991 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.215582 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.215660 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.215669 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:15.215993 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 19:05:15.216033 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:05:15.216763 8087 raft_consensus.cc:2454] T 62a8114d15154ba298d6cee5bb4c7097 P 4e5e4690e8104ca09a7592a858d2983c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 667b4a82651c4a26a717f37627d060d3 in term 1.
I20260925 19:05:15.215965 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:15.212374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:34699 (local address 127.0.196.195:50905)
0925 19:05:15.212723 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:15.212737 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:15.212793 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:15.212950 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:15.212951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:15.212953 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:15.213008 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:15.213010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.213923 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.213924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:15.214911 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:15.214913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.215700 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.215701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:15.215790 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.215794 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:05:15.215845 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 19:05:15.215887 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329}
I20260925 19:05:15.216912 8173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:15.212569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:39543 (local address 127.0.196.194:43897)
0925 19:05:15.212868 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:15.212871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:15.212878 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:05:15.212897 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:15.212898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:15.212906 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:15.213114 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:15.213150 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.214397 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.215816 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.216486 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.216602 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.216611 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:15.216787 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 19:05:15.216825 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":119,"threads_started":1}
I20260925 19:05:15.216919 8139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:15.212354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:43897 (local address 127.0.196.195:39543)
0925 19:05:15.212444 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:15.212456 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:15.212499 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:15.212994 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:15.212995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:15.212995 (+ 0us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:15.213044 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:15.213045 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.214503 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.214504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:15.215737 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:15.215739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.216603 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.216604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:15.216683 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.216687 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:05:15.216778 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 19:05:15.216823 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 19:05:15.216953 7748 leader_election.cc:302] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e5e4690e8104ca09a7592a858d2983c, 667b4a82651c4a26a717f37627d060d3; no voters:
I20260925 19:05:15.217167 7815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a8114d15154ba298d6cee5bb4c7097" candidate_uuid: "4e5e4690e8104ca09a7592a858d2983c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "667b4a82651c4a26a717f37627d060d3" is_pre_election: true
I20260925 19:05:15.217218 7951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a8114d15154ba298d6cee5bb4c7097" candidate_uuid: "667b4a82651c4a26a717f37627d060d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a29d1c1c84644fea8558af4f2c9823f"
I20260925 19:05:15.217298 7815 raft_consensus.cc:2379] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e5e4690e8104ca09a7592a858d2983c in current term 1: Already voted for candidate 667b4a82651c4a26a717f37627d060d3 in this term.
I20260925 19:05:15.217303 7951 raft_consensus.cc:3037] T 62a8114d15154ba298d6cee5bb4c7097 P 6a29d1c1c84644fea8558af4f2c9823f [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:15.217500 8174 raft_consensus.cc:2781] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:05:15.217653 8174 raft_consensus.cc:686] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3 [term 1 LEADER]: Becoming Leader. State: Replica: 667b4a82651c4a26a717f37627d060d3, State: Running, Role: LEADER
I20260925 19:05:15.217760 8174 consensus_queue.cc:237] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b4a82651c4a26a717f37627d060d3" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34699 } } peers { permanent_uuid: "4e5e4690e8104ca09a7592a858d2983c" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 36681 } } peers { permanent_uuid: "6a29d1c1c84644fea8558af4f2c9823f" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43897 } }
I20260925 19:05:15.218207 7951 raft_consensus.cc:2454] T 62a8114d15154ba298d6cee5bb4c7097 P 6a29d1c1c84644fea8558af4f2c9823f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 667b4a82651c4a26a717f37627d060d3 in term 1.
I20260925 19:05:15.217296 7950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a8114d15154ba298d6cee5bb4c7097" candidate_uuid: "4e5e4690e8104ca09a7592a858d2983c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a29d1c1c84644fea8558af4f2c9823f" is_pre_election: true
I20260925 19:05:15.218405 7950 raft_consensus.cc:2379] T 62a8114d15154ba298d6cee5bb4c7097 P 6a29d1c1c84644fea8558af4f2c9823f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e5e4690e8104ca09a7592a858d2983c in current term 1: Already voted for candidate 667b4a82651c4a26a717f37627d060d3 in this term.
I20260925 19:05:15.218686 8021 leader_election.cc:302] T 62a8114d15154ba298d6cee5bb4c7097 P 4e5e4690e8104ca09a7592a858d2983c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e5e4690e8104ca09a7592a858d2983c; no voters: 667b4a82651c4a26a717f37627d060d3, 6a29d1c1c84644fea8558af4f2c9823f
I20260925 19:05:15.218741 6896 catalog_manager.cc:5495] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 667b4a82651c4a26a717f37627d060d3 (127.0.196.193). New cstate: current_term: 1 leader_uuid: "667b4a82651c4a26a717f37627d060d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667b4a82651c4a26a717f37627d060d3" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e5e4690e8104ca09a7592a858d2983c" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a29d1c1c84644fea8558af4f2c9823f" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43897 } health_report { overall_health: UNKNOWN } } }
I20260925 19:05:15.219113 8170 leader_election.cc:288] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3 [CANDIDATE]: Term 1 election: Requested vote from peers 4e5e4690e8104ca09a7592a858d2983c (127.0.196.195:36681), 6a29d1c1c84644fea8558af4f2c9823f (127.0.196.194:43897)
I20260925 19:05:15.219424 8172 raft_consensus.cc:2726] T 62a8114d15154ba298d6cee5bb4c7097 P 4e5e4690e8104ca09a7592a858d2983c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 19:05:15.230113 8161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:15.227772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.0.196.193:34699)
0925 19:05:15.227842 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:15.227844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:15.227853 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:05:15.227904 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:15.227904 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:15.227921 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:15.227981 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:05:15.228086 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.228564 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.229408 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.229585 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.229638 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.229798 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:05:15.229902 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:05:15.229964 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:15.230002 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 19:05:15.230018 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 19:05:15.232589 7950 raft_consensus.cc:1261] T 62a8114d15154ba298d6cee5bb4c7097 P 6a29d1c1c84644fea8558af4f2c9823f [term 1 FOLLOWER]: Refusing update from remote peer 667b4a82651c4a26a717f37627d060d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:05:15.232640 8087 raft_consensus.cc:1261] T 62a8114d15154ba298d6cee5bb4c7097 P 4e5e4690e8104ca09a7592a858d2983c [term 1 FOLLOWER]: Refusing update from remote peer 667b4a82651c4a26a717f37627d060d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:05:15.232944 8174 consensus_queue.cc:1035] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a29d1c1c84644fea8558af4f2c9823f" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:05:15.233063 8170 consensus_queue.cc:1035] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e5e4690e8104ca09a7592a858d2983c" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:05:15.236680 8180 mvcc.cc:205] Tried to move back new op lower bound from 7333327319989727232 to 7333327319932547072. Current Snapshot: MvccSnapshot[applied={T|T < 7333327319989727232}]
I20260925 19:05:15.237840 8178 mvcc.cc:205] Tried to move back new op lower bound from 7333327319989727232 to 7333327319932547072. Current Snapshot: MvccSnapshot[applied={T|T < 7333327319989727232 or (T in {7333327319989727232})}]
I20260925 19:05:15.239776 6896 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:05:15.239864 6896 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:05:15.241206 6896 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c9e241c502f4fed8aae1e80224e05b9: Sending DeleteTablet for 3 replicas of tablet 62a8114d15154ba298d6cee5bb4c7097
I20260925 19:05:15.241765 7794 tablet_service.cc:1514] Processing DeleteTablet for tablet 62a8114d15154ba298d6cee5bb4c7097 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:05:15 UTC) from {username='oryx', principal='oryx/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:32926
I20260925 19:05:15.241813 7931 tablet_service.cc:1514] Processing DeleteTablet for tablet 62a8114d15154ba298d6cee5bb4c7097 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:05:15 UTC) from {username='oryx', principal='oryx/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:59904
I20260925 19:05:15.242183 8067 tablet_service.cc:1514] Processing DeleteTablet for tablet 62a8114d15154ba298d6cee5bb4c7097 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:05:15 UTC) from {username='oryx', principal='oryx/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:45332
I20260925 19:05:15.242419 8188 tablet_replica.cc:330] T 62a8114d15154ba298d6cee5bb4c7097 P 6a29d1c1c84644fea8558af4f2c9823f: stopping tablet replica
I20260925 19:05:15.242534 8188 raft_consensus.cc:2229] T 62a8114d15154ba298d6cee5bb4c7097 P 6a29d1c1c84644fea8558af4f2c9823f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:05:15.242707 8188 raft_consensus.cc:2258] T 62a8114d15154ba298d6cee5bb4c7097 P 6a29d1c1c84644fea8558af4f2c9823f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:05:15.242765 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 6864
I20260925 19:05:15.242851 8187 tablet_replica.cc:330] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3: stopping tablet replica
I20260925 19:05:15.242976 8187 raft_consensus.cc:2229] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:05:15.243232 8187 raft_consensus.cc:2258] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:05:15.243741 8187 ts_tablet_manager.cc:1905] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:05:15.243881 8188 ts_tablet_manager.cc:1905] T 62a8114d15154ba298d6cee5bb4c7097 P 6a29d1c1c84644fea8558af4f2c9823f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:05:15.244867 8189 tablet_replica.cc:330] T 62a8114d15154ba298d6cee5bb4c7097 P 4e5e4690e8104ca09a7592a858d2983c: stopping tablet replica
I20260925 19:05:15.245544 8187 ts_tablet_manager.cc:1918] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:05:15.245685 8187 log.cc:1196] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-0/wal/wals/62a8114d15154ba298d6cee5bb4c7097
I20260925 19:05:15.244987 8189 raft_consensus.cc:2229] T 62a8114d15154ba298d6cee5bb4c7097 P 4e5e4690e8104ca09a7592a858d2983c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:05:15.245999 8189 raft_consensus.cc:2258] T 62a8114d15154ba298d6cee5bb4c7097 P 4e5e4690e8104ca09a7592a858d2983c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:05:15.246146 8187 ts_tablet_manager.cc:1939] T 62a8114d15154ba298d6cee5bb4c7097 P 667b4a82651c4a26a717f37627d060d3: Deleting consensus metadata
I20260925 19:05:15.246376 8188 ts_tablet_manager.cc:1918] T 62a8114d15154ba298d6cee5bb4c7097 P 6a29d1c1c84644fea8558af4f2c9823f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:05:15.246449 8188 log.cc:1196] T 62a8114d15154ba298d6cee5bb4c7097 P 6a29d1c1c84644fea8558af4f2c9823f: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/wal/wals/62a8114d15154ba298d6cee5bb4c7097
I20260925 19:05:15.246726 8188 ts_tablet_manager.cc:1939] T 62a8114d15154ba298d6cee5bb4c7097 P 6a29d1c1c84644fea8558af4f2c9823f: Deleting consensus metadata
W20260925 19:05:15.249202 7884 connection.cc:535] server connection from 127.0.0.1:59904 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 19:05:15.249459 7751 connection.cc:535] server connection from 127.0.0.1:32926 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 19:05:15.249609 8189 ts_tablet_manager.cc:1905] T 62a8114d15154ba298d6cee5bb4c7097 P 4e5e4690e8104ca09a7592a858d2983c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:05:15.251673 8189 ts_tablet_manager.cc:1918] T 62a8114d15154ba298d6cee5bb4c7097 P 4e5e4690e8104ca09a7592a858d2983c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:05:15.251766 8189 log.cc:1196] T 62a8114d15154ba298d6cee5bb4c7097 P 4e5e4690e8104ca09a7592a858d2983c: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/wal/wals/62a8114d15154ba298d6cee5bb4c7097
I20260925 19:05:15.252085 8189 ts_tablet_manager.cc:1939] T 62a8114d15154ba298d6cee5bb4c7097 P 4e5e4690e8104ca09a7592a858d2983c: Deleting consensus metadata
W20260925 19:05:15.252458 8023 connection.cc:441] server connection from 127.0.0.1:45332 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45332 (request call id 1) could send its response
I20260925 19:05:15.259815 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:15.251534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41998 (local address 127.0.196.253:36251)
0925 19:05:15.251578 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:15.251580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:15.251771 (+ 191us) server_negotiation.cc:407] Connection header received
0925 19:05:15.251895 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:15.251896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:15.251908 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:15.251952 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:15.252805 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.253298 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.254090 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.254306 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.254407 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.256498 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:15.256499 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:15.256500 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:15.256508 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:15.258170 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:15.258993 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:15.258994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:15.258995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:15.259061 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:15.259263 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:15.259264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:15.259264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:15.259387 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:15.259493 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:15.259600 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 19:05:15.259693 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:05:15.260555 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:15.251479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59354 (local address 127.0.196.252:33081)
0925 19:05:15.251532 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:15.251534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:15.251675 (+ 141us) server_negotiation.cc:407] Connection header received
0925 19:05:15.251877 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:15.251878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:15.251891 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:15.251923 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:15.253062 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.253899 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.255181 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:15.255525 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:15.255615 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:15.257251 (+ 1636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:15.257253 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:15.257253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:15.257261 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:15.258981 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:15.259390 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:15.259391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:15.259391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:15.259447 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:15.259663 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:15.259663 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:15.259664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:15.259797 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:15.259868 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:15.260421 (+ 553us) server_negotiation.cc:299] Negotiation successful
0925 19:05:15.260495 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 19:05:16.238214 8197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:16.237840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.196.194:51189)
0925 19:05:16.238004 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:16.238096 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":79,"threads_started":1}
I20260925 19:05:16.238389 8198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:16.238008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.196.193:47077)
0925 19:05:16.238268 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:16.238337 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":55,"threads_started":1}
W20260925 19:05:16.238591 7998 heartbeater.cc:644] Failed to heartbeat to 127.0.196.254:38655 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
W20260925 19:05:16.238873 7862 heartbeater.cc:644] Failed to heartbeat to 127.0.196.254:38655 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
I20260925 19:05:16.238931 8199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:16.238472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.196.195:49007)
0925 19:05:16.238787 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:16.238884 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":51,"threads_started":1}
W20260925 19:05:16.239114 8134 heartbeater.cc:644] Failed to heartbeat to 127.0.196.254:38655 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
I20260925 19:05:16.741837 8200 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2c9e241c502f4fed8aae1e80224e05b9)
I20260925 19:05:16.742062 8200 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: false } }
I20260925 19:05:16.742192 8201 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2c9e241c502f4fed8aae1e80224e05b9)
I20260925 19:05:16.742347 8201 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: false } }
I20260925 19:05:16.742765 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:16.742531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.0.1:42680)
0925 19:05:16.742661 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:16.742717 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":48,"threads_started":1}
W20260925 19:05:16.743125 7478 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
I20260925 19:05:16.743157 8203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:16.742751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.0.1:42696)
0925 19:05:16.743073 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:16.743119 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":47,"threads_started":1}
W20260925 19:05:16.743250 7664 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
I20260925 19:05:16.743410 8201 leader_election.cc:288] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2c9e241c502f4fed8aae1e80224e05b9 (127.0.196.254:38655), fc40919e46b24cbebc864883e6aa8e29 (127.0.196.253:36251)
I20260925 19:05:16.743451 8200 leader_election.cc:288] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2c9e241c502f4fed8aae1e80224e05b9 (127.0.196.254:38655), dfdd31dd8df548429c1f60d15fc974fa (127.0.196.252:33081)
I20260925 19:05:16.743772 8204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:16.743675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.0.1:42698)
0925 19:05:16.743710 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:16.743745 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 19:05:16.743804 8203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:16.743717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.0.1:42706)
0925 19:05:16.743754 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:16.743779 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 19:05:16.743861 7664 leader_election.cc:334] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2c9e241c502f4fed8aae1e80224e05b9 (127.0.196.254:38655): Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
W20260925 19:05:16.744056 7478 leader_election.cc:334] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2c9e241c502f4fed8aae1e80224e05b9 (127.0.196.254:38655): Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
I20260925 19:05:16.746171 8205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:16.742933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:36251 (local address 127.0.0.1:42000)
0925 19:05:16.743188 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:16.743198 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:16.743281 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:16.743474 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:16.743475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:16.743477 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:16.743536 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:16.743538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:16.744460 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:16.744461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:16.745312 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:16.745314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:16.745909 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:16.745910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:16.745977 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:16.745981 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:05:16.746055 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 19:05:16.746102 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":44,"threads_started":1}
I20260925 19:05:16.746209 8207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:16.743049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42000 (local address 127.0.196.253:36251)
0925 19:05:16.743147 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:16.743149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:16.743242 (+ 93us) server_negotiation.cc:407] Connection header received
0925 19:05:16.743358 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:16.743359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:16.743369 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:16.743404 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:16.743619 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:16.744332 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:16.745411 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:16.745813 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:16.745905 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:16.745917 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:16.746063 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 19:05:16.746139 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":71,"threads_started":1}
I20260925 19:05:16.746274 8206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:16.742982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59360 (local address 127.0.196.252:33081)
0925 19:05:16.743451 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:16.743463 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:16.743472 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:05:16.743495 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:16.743496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:16.743505 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:16.743750 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:16.743781 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:16.744386 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:16.745489 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:16.745867 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:16.745946 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:16.745956 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:16.746124 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 19:05:16.746167 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":53,"threads_started":1}
I20260925 19:05:16.746536 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:16.742759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:33081 (local address 127.0.0.1:59360)
0925 19:05:16.742961 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:16.742972 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:16.743031 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:16.743623 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:16.743624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:16.743627 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:16.743683 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:16.743685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:16.744469 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:16.744471 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:16.745390 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:16.745393 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:16.745956 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:16.745957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:16.746018 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:16.746022 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:05:16.746105 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 19:05:16.746150 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":1,"thread_start_us":60,"threads_started":1}
I20260925 19:05:16.746644 7504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dfdd31dd8df548429c1f60d15fc974fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fc40919e46b24cbebc864883e6aa8e29" is_pre_election: true
I20260925 19:05:16.746758 7504 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfdd31dd8df548429c1f60d15fc974fa in term 1.
I20260925 19:05:16.746790 7690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc40919e46b24cbebc864883e6aa8e29" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dfdd31dd8df548429c1f60d15fc974fa" is_pre_election: true
I20260925 19:05:16.746894 7690 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc40919e46b24cbebc864883e6aa8e29 in term 1.
I20260925 19:05:16.746912 7667 leader_election.cc:302] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dfdd31dd8df548429c1f60d15fc974fa, fc40919e46b24cbebc864883e6aa8e29; no voters: 2c9e241c502f4fed8aae1e80224e05b9
I20260925 19:05:16.747010 8201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 19:05:16.747082 8201 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2c9e241c502f4fed8aae1e80224e05b9)
I20260925 19:05:16.747124 8201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [term 1 FOLLOWER]: Advancing to term 2
I20260925 19:05:16.747119 7479 leader_election.cc:302] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dfdd31dd8df548429c1f60d15fc974fa, fc40919e46b24cbebc864883e6aa8e29; no voters: 2c9e241c502f4fed8aae1e80224e05b9
I20260925 19:05:16.747211 8200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 19:05:16.747248 8200 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2c9e241c502f4fed8aae1e80224e05b9)
I20260925 19:05:16.747279 8200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 1 FOLLOWER]: Advancing to term 2
I20260925 19:05:16.747849 8201 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: false } }
I20260925 19:05:16.747983 8201 leader_election.cc:288] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [CANDIDATE]: Term 2 election: Requested vote from peers 2c9e241c502f4fed8aae1e80224e05b9 (127.0.196.254:38655), fc40919e46b24cbebc864883e6aa8e29 (127.0.196.253:36251)
I20260925 19:05:16.747958 8200 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: false } }
I20260925 19:05:16.748109 7504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dfdd31dd8df548429c1f60d15fc974fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fc40919e46b24cbebc864883e6aa8e29"
I20260925 19:05:16.748140 8205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:16.748053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.0.1:42722)
0925 19:05:16.748083 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:16.748116 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260925 19:05:16.748168 8200 leader_election.cc:288] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [CANDIDATE]: Term 2 election: Requested vote from peers 2c9e241c502f4fed8aae1e80224e05b9 (127.0.196.254:38655), dfdd31dd8df548429c1f60d15fc974fa (127.0.196.252:33081)
I20260925 19:05:16.748183 7504 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate dfdd31dd8df548429c1f60d15fc974fa in current term 2: Already voted for candidate fc40919e46b24cbebc864883e6aa8e29 in this term.
I20260925 19:05:16.748334 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:16.748235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.0.1:42734)
0925 19:05:16.748286 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:16.748310 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 19:05:16.748358 7690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc40919e46b24cbebc864883e6aa8e29" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dfdd31dd8df548429c1f60d15fc974fa"
I20260925 19:05:16.748440 7690 raft_consensus.cc:2379] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fc40919e46b24cbebc864883e6aa8e29 in current term 2: Already voted for candidate dfdd31dd8df548429c1f60d15fc974fa in this term.
I20260925 19:05:16.748627 8205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:16.748540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.0.1:42744)
0925 19:05:16.748575 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:16.748605 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260925 19:05:16.748736 7664 leader_election.cc:334] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2c9e241c502f4fed8aae1e80224e05b9 (127.0.196.254:38655): Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
I20260925 19:05:16.748780 7664 leader_election.cc:302] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dfdd31dd8df548429c1f60d15fc974fa; no voters: 2c9e241c502f4fed8aae1e80224e05b9, fc40919e46b24cbebc864883e6aa8e29
I20260925 19:05:16.748843 8201 raft_consensus.cc:2726] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 19:05:16.748951 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:16.748660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.0.1:42748)
0925 19:05:16.748893 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:16.748921 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217}
W20260925 19:05:16.749029 7478 leader_election.cc:334] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2c9e241c502f4fed8aae1e80224e05b9 (127.0.196.254:38655): Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
I20260925 19:05:16.749081 7478 leader_election.cc:302] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc40919e46b24cbebc864883e6aa8e29; no voters: 2c9e241c502f4fed8aae1e80224e05b9, dfdd31dd8df548429c1f60d15fc974fa
I20260925 19:05:16.749156 8200 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 19:05:17.239504 8211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:17.239187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.196.193:53343)
0925 19:05:17.239395 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:17.239441 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":77,"threads_started":1}
I20260925 19:05:17.239626 8210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:17.239187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.196.194:49349)
0925 19:05:17.239526 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:17.239565 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":80,"threads_started":1}
I20260925 19:05:17.239826 8212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:17.239500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.196.195:37863)
0925 19:05:17.239748 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:17.239783 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":77,"threads_started":1}
I20260925 19:05:18.240427 8215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.240228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.196.195:60287)
0925 19:05:18.240331 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:18.240370 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":45,"threads_started":1}
I20260925 19:05:18.240531 8214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.240180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.196.193:36687)
0925 19:05:18.240441 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:18.240483 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":65,"threads_started":1}
I20260925 19:05:18.240715 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.240184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.196.194:55585)
0925 19:05:18.240609 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:18.240670 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":64,"threads_started":1}
I20260925 19:05:18.640765 8216 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:05:18.640903 8216 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: false } }
I20260925 19:05:18.641104 8216 leader_election.cc:288] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2c9e241c502f4fed8aae1e80224e05b9 (127.0.196.254:38655), dfdd31dd8df548429c1f60d15fc974fa (127.0.196.252:33081)
I20260925 19:05:18.641367 7690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc40919e46b24cbebc864883e6aa8e29" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dfdd31dd8df548429c1f60d15fc974fa" is_pre_election: true
I20260925 19:05:18.641453 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.641237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.0.1:43042)
0925 19:05:18.641362 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:18.641410 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260925 19:05:18.641485 7690 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc40919e46b24cbebc864883e6aa8e29 in term 2.
I20260925 19:05:18.641654 7479 leader_election.cc:302] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dfdd31dd8df548429c1f60d15fc974fa, fc40919e46b24cbebc864883e6aa8e29; no voters:
I20260925 19:05:18.641775 8216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 19:05:18.641873 8216 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:05:18.641911 8216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 2 FOLLOWER]: Advancing to term 3
I20260925 19:05:18.641944 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.641812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.0.1:43046)
0925 19:05:18.641866 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:18.641900 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 19:05:18.642063 7478 leader_election.cc:334] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2c9e241c502f4fed8aae1e80224e05b9 (127.0.196.254:38655): Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
I20260925 19:05:18.642722 8216 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: false } }
I20260925 19:05:18.642879 8216 leader_election.cc:288] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [CANDIDATE]: Term 3 election: Requested vote from peers 2c9e241c502f4fed8aae1e80224e05b9 (127.0.196.254:38655), dfdd31dd8df548429c1f60d15fc974fa (127.0.196.252:33081)
I20260925 19:05:18.643043 7690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc40919e46b24cbebc864883e6aa8e29" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dfdd31dd8df548429c1f60d15fc974fa"
I20260925 19:05:18.643131 7690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [term 2 FOLLOWER]: Advancing to term 3
I20260925 19:05:18.643119 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.643008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.0.1:43048)
0925 19:05:18.643065 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:18.643093 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 19:05:18.643433 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.643382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.0.1:43050)
0925 19:05:18.643403 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:18.643421 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260925 19:05:18.643522 7478 leader_election.cc:334] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 2c9e241c502f4fed8aae1e80224e05b9 (127.0.196.254:38655): Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
I20260925 19:05:18.643946 7690 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc40919e46b24cbebc864883e6aa8e29 in term 3.
I20260925 19:05:18.644337 7479 leader_election.cc:302] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dfdd31dd8df548429c1f60d15fc974fa, fc40919e46b24cbebc864883e6aa8e29; no voters: 2c9e241c502f4fed8aae1e80224e05b9
I20260925 19:05:18.644440 8216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 19:05:18.644589 8216 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [term 3 LEADER]: Becoming Leader. State: Replica: fc40919e46b24cbebc864883e6aa8e29, State: Running, Role: LEADER
I20260925 19:05:18.644798 8216 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: false } }
I20260925 19:05:18.645170 8219 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc40919e46b24cbebc864883e6aa8e29. Latest consensus state: current_term: 3 leader_uuid: "fc40919e46b24cbebc864883e6aa8e29" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: false } } }
I20260925 19:05:18.645296 8219 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:18.645529 8221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:05:18.646000 8221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:05:18.646093 8221 catalog_manager.cc:1259] Loaded cluster ID: 1bd3a743ba4c4325831d9b5ff9980438
I20260925 19:05:18.646143 8221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:05:18.646384 8221 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:05:18.646466 8221 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29: Loaded TSK: 0
I20260925 19:05:18.646610 8221 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:05:18.667302 7493 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41998:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:05:18.668996 7493 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:05:18.670214 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.670096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.0.1:43060)
0925 19:05:18.670148 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:18.670187 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 19:05:18.670267 7690 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [term 3 FOLLOWER]: Refusing update from remote peer fc40919e46b24cbebc864883e6aa8e29: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260925 19:05:18.670521 8219 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 19:05:18.670585 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.670462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.0.1:43062)
0925 19:05:18.670501 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:18.670531 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 19:05:18.670717 7478 consensus_peers.cc:487] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 -> Peer 2c9e241c502f4fed8aae1e80224e05b9 (127.0.196.254:38655): Couldn't send request to peer 2c9e241c502f4fed8aae1e80224e05b9. Status: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 19:05:18.671181 8223 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc40919e46b24cbebc864883e6aa8e29. Latest consensus state: current_term: 3 leader_uuid: "fc40919e46b24cbebc864883e6aa8e29" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: false } } }
I20260925 19:05:18.671307 8223 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:05:18.671777 8216 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "fc40919e46b24cbebc864883e6aa8e29" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: false } } }
I20260925 19:05:18.671862 8216 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:18.672011 8223 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "fc40919e46b24cbebc864883e6aa8e29" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2c9e241c502f4fed8aae1e80224e05b9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38655 } } peers { permanent_uuid: "fc40919e46b24cbebc864883e6aa8e29" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 36251 } attrs { promote: false } } peers { permanent_uuid: "dfdd31dd8df548429c1f60d15fc974fa" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 33081 } attrs { promote: false } } }
I20260925 19:05:18.672097 8223 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfdd31dd8df548429c1f60d15fc974fa [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:05:18.677479 8231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.674035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59916 (local address 127.0.196.194:43897)
0925 19:05:18.674206 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:18.674208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:18.674376 (+ 168us) server_negotiation.cc:407] Connection header received
0925 19:05:18.674498 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:18.674499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:18.674511 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:18.674558 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:18.674756 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:18.675236 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:18.676717 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:18.677105 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:18.677189 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:18.677202 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:18.677345 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 19:05:18.677384 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":55,"threads_started":1}
I20260925 19:05:18.677587 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.673910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:36681 (local address 127.0.0.1:45334)
0925 19:05:18.674089 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:18.674103 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:18.674175 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:18.674505 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:18.674506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:18.674508 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:18.674561 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:18.674563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:18.675491 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:18.675491 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:18.676608 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:18.676611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:18.677358 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:18.677359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:18.677413 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:18.677416 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:05:18.677444 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 19:05:18.677478 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":30}
I20260925 19:05:18.677626 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.674304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32946 (local address 127.0.196.193:34699)
0925 19:05:18.674411 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:18.674413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:18.674420 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:05:18.674446 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:18.674447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:18.674458 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:18.674543 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:18.674740 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:18.675458 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:18.676559 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:18.677142 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:18.677211 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:18.677224 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:18.677510 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 19:05:18.677559 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":55,"threads_started":1}
I20260925 19:05:18.677683 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.673992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45334 (local address 127.0.196.195:36681)
0925 19:05:18.674105 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:18.674107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:18.674313 (+ 206us) server_negotiation.cc:407] Connection header received
0925 19:05:18.674343 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:18.674345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:18.674412 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:18.674504 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:18.674645 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:18.675412 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:18.676691 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:18.677275 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:18.677537 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:18.677549 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:18.677567 (+ 18us) server_negotiation.cc:299] Negotiation successful
0925 19:05:18.677607 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":59,"threads_started":1}
I20260925 19:05:18.677487 8232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.673967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:43897 (local address 127.0.0.1:59916)
0925 19:05:18.674258 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:18.674268 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:18.674403 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:18.674640 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:18.674641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:18.674642 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:18.674670 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:18.674671 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:18.675331 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:18.675332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:18.676643 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:18.676644 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:18.677198 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:18.677199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:18.677262 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:18.677266 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:05:18.677341 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 19:05:18.677389 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":35,"thread_start_us":67,"threads_started":1}
I20260925 19:05:18.677801 8229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.674142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:34699 (local address 127.0.0.1:32946)
0925 19:05:18.674264 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:18.674274 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:18.674319 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:18.674592 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:18.674593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:18.674595 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:18.674666 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:18.674668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:18.675544 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:18.675545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:18.676478 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:18.676480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:18.677259 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:18.677260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:18.677328 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:18.677334 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 19:05:18.677394 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 19:05:18.677445 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
I20260925 19:05:18.678287 7794 tablet_service.cc:1467] Processing CreateTablet for tablet ea0144686a2442ad83097220e178b7c4 (DEFAULT_TABLE table=test-table [id=0e1e5d1efa7f4281b143df7a098dcece]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:05:18.678360 8067 tablet_service.cc:1467] Processing CreateTablet for tablet ea0144686a2442ad83097220e178b7c4 (DEFAULT_TABLE table=test-table [id=0e1e5d1efa7f4281b143df7a098dcece]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:05:18.678442 7931 tablet_service.cc:1467] Processing CreateTablet for tablet ea0144686a2442ad83097220e178b7c4 (DEFAULT_TABLE table=test-table [id=0e1e5d1efa7f4281b143df7a098dcece]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:05:18.678467 7794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea0144686a2442ad83097220e178b7c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:18.678500 8067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea0144686a2442ad83097220e178b7c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:18.678537 7931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea0144686a2442ad83097220e178b7c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:18.679870 8234 tablet_bootstrap.cc:492] T ea0144686a2442ad83097220e178b7c4 P 667b4a82651c4a26a717f37627d060d3: Bootstrap starting.
I20260925 19:05:18.680143 8236 tablet_bootstrap.cc:492] T ea0144686a2442ad83097220e178b7c4 P 6a29d1c1c84644fea8558af4f2c9823f: Bootstrap starting.
I20260925 19:05:18.680318 8234 tablet_bootstrap.cc:654] T ea0144686a2442ad83097220e178b7c4 P 667b4a82651c4a26a717f37627d060d3: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:18.680519 8236 tablet_bootstrap.cc:654] T ea0144686a2442ad83097220e178b7c4 P 6a29d1c1c84644fea8558af4f2c9823f: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:18.680828 8235 tablet_bootstrap.cc:492] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c: Bootstrap starting.
I20260925 19:05:18.680931 8234 tablet_bootstrap.cc:492] T ea0144686a2442ad83097220e178b7c4 P 667b4a82651c4a26a717f37627d060d3: No bootstrap required, opened a new log
I20260925 19:05:18.681007 8234 ts_tablet_manager.cc:1397] T ea0144686a2442ad83097220e178b7c4 P 667b4a82651c4a26a717f37627d060d3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:18.681097 8236 tablet_bootstrap.cc:492] T ea0144686a2442ad83097220e178b7c4 P 6a29d1c1c84644fea8558af4f2c9823f: No bootstrap required, opened a new log
I20260925 19:05:18.681155 8236 ts_tablet_manager.cc:1397] T ea0144686a2442ad83097220e178b7c4 P 6a29d1c1c84644fea8558af4f2c9823f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:18.681126 8234 raft_consensus.cc:348] T ea0144686a2442ad83097220e178b7c4 P 667b4a82651c4a26a717f37627d060d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5e4690e8104ca09a7592a858d2983c" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 36681 } } peers { permanent_uuid: "6a29d1c1c84644fea8558af4f2c9823f" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43897 } } peers { permanent_uuid: "667b4a82651c4a26a717f37627d060d3" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34699 } }
I20260925 19:05:18.681222 8235 tablet_bootstrap.cc:654] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:18.681200 8234 raft_consensus.cc:374] T ea0144686a2442ad83097220e178b7c4 P 667b4a82651c4a26a717f37627d060d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:18.681284 8236 raft_consensus.cc:348] T ea0144686a2442ad83097220e178b7c4 P 6a29d1c1c84644fea8558af4f2c9823f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5e4690e8104ca09a7592a858d2983c" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 36681 } } peers { permanent_uuid: "6a29d1c1c84644fea8558af4f2c9823f" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43897 } } peers { permanent_uuid: "667b4a82651c4a26a717f37627d060d3" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34699 } }
I20260925 19:05:18.681340 8234 raft_consensus.cc:729] T ea0144686a2442ad83097220e178b7c4 P 667b4a82651c4a26a717f37627d060d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 667b4a82651c4a26a717f37627d060d3, State: Initialized, Role: FOLLOWER
I20260925 19:05:18.681372 8236 raft_consensus.cc:374] T ea0144686a2442ad83097220e178b7c4 P 6a29d1c1c84644fea8558af4f2c9823f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:18.681399 8236 raft_consensus.cc:729] T ea0144686a2442ad83097220e178b7c4 P 6a29d1c1c84644fea8558af4f2c9823f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a29d1c1c84644fea8558af4f2c9823f, State: Initialized, Role: FOLLOWER
I20260925 19:05:18.681474 8236 consensus_queue.cc:260] T ea0144686a2442ad83097220e178b7c4 P 6a29d1c1c84644fea8558af4f2c9823f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5e4690e8104ca09a7592a858d2983c" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 36681 } } peers { permanent_uuid: "6a29d1c1c84644fea8558af4f2c9823f" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43897 } } peers { permanent_uuid: "667b4a82651c4a26a717f37627d060d3" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34699 } }
I20260925 19:05:18.681470 8234 consensus_queue.cc:260] T ea0144686a2442ad83097220e178b7c4 P 667b4a82651c4a26a717f37627d060d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5e4690e8104ca09a7592a858d2983c" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 36681 } } peers { permanent_uuid: "6a29d1c1c84644fea8558af4f2c9823f" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43897 } } peers { permanent_uuid: "667b4a82651c4a26a717f37627d060d3" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34699 } }
I20260925 19:05:18.681674 8234 ts_tablet_manager.cc:1428] T ea0144686a2442ad83097220e178b7c4 P 667b4a82651c4a26a717f37627d060d3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 19:05:18.681684 8236 ts_tablet_manager.cc:1428] T ea0144686a2442ad83097220e178b7c4 P 6a29d1c1c84644fea8558af4f2c9823f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 19:05:18.682052 8214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.681894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.196.193:51957)
0925 19:05:18.681981 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:18.682017 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 19:05:18.682353 7999 heartbeater.cc:503] Master 127.0.196.253:36251 was elected leader, sending a full tablet report...
I20260925 19:05:18.682556 8235 tablet_bootstrap.cc:492] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c: No bootstrap required, opened a new log
I20260925 19:05:18.682607 8235 ts_tablet_manager.cc:1397] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:05:18.682765 8235 raft_consensus.cc:348] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5e4690e8104ca09a7592a858d2983c" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 36681 } } peers { permanent_uuid: "6a29d1c1c84644fea8558af4f2c9823f" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43897 } } peers { permanent_uuid: "667b4a82651c4a26a717f37627d060d3" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34699 } }
I20260925 19:05:18.682837 8235 raft_consensus.cc:374] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:18.682858 8235 raft_consensus.cc:729] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e5e4690e8104ca09a7592a858d2983c, State: Initialized, Role: FOLLOWER
I20260925 19:05:18.682904 8235 consensus_queue.cc:260] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5e4690e8104ca09a7592a858d2983c" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 36681 } } peers { permanent_uuid: "6a29d1c1c84644fea8558af4f2c9823f" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43897 } } peers { permanent_uuid: "667b4a82651c4a26a717f37627d060d3" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34699 } }
I20260925 19:05:18.683065 8235 ts_tablet_manager.cc:1428] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:18.683214 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.682359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.196.194:42043)
0925 19:05:18.683149 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:18.683184 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":743}
I20260925 19:05:18.683414 8215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.683289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.196.195:58465)
0925 19:05:18.683362 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:18.683391 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 19:05:18.683866 8135 heartbeater.cc:503] Master 127.0.196.253:36251 was elected leader, sending a full tablet report...
I20260925 19:05:18.683848 7863 heartbeater.cc:503] Master 127.0.196.253:36251 was elected leader, sending a full tablet report...
I20260925 19:05:18.683911 8214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.683743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.196.193:55895)
0925 19:05:18.683798 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:18.683831 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260925 19:05:18.684248 8215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.684102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.196.195:35153)
0925 19:05:18.684197 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:18.684223 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 19:05:18.686564 8242 raft_consensus.cc:482] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:05:18.686685 8242 raft_consensus.cc:504] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5e4690e8104ca09a7592a858d2983c" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 36681 } } peers { permanent_uuid: "6a29d1c1c84644fea8558af4f2c9823f" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43897 } } peers { permanent_uuid: "667b4a82651c4a26a717f37627d060d3" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34699 } }
I20260925 19:05:18.686861 8242 leader_election.cc:288] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a29d1c1c84644fea8558af4f2c9823f (127.0.196.194:43897), 667b4a82651c4a26a717f37627d060d3 (127.0.196.193:34699)
I20260925 19:05:18.687036 7815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea0144686a2442ad83097220e178b7c4" candidate_uuid: "4e5e4690e8104ca09a7592a858d2983c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "667b4a82651c4a26a717f37627d060d3" is_pre_election: true
I20260925 19:05:18.687137 7815 raft_consensus.cc:2454] T ea0144686a2442ad83097220e178b7c4 P 667b4a82651c4a26a717f37627d060d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e5e4690e8104ca09a7592a858d2983c in term 0.
I20260925 19:05:18.687175 7950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea0144686a2442ad83097220e178b7c4" candidate_uuid: "4e5e4690e8104ca09a7592a858d2983c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a29d1c1c84644fea8558af4f2c9823f" is_pre_election: true
I20260925 19:05:18.687268 8023 leader_election.cc:302] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e5e4690e8104ca09a7592a858d2983c, 667b4a82651c4a26a717f37627d060d3; no voters:
I20260925 19:05:18.687279 7950 raft_consensus.cc:2454] T ea0144686a2442ad83097220e178b7c4 P 6a29d1c1c84644fea8558af4f2c9823f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e5e4690e8104ca09a7592a858d2983c in term 0.
I20260925 19:05:18.687366 8242 raft_consensus.cc:2781] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:05:18.687425 8242 raft_consensus.cc:482] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:05:18.687461 8242 raft_consensus.cc:3037] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:18.688127 8242 raft_consensus.cc:504] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5e4690e8104ca09a7592a858d2983c" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 36681 } } peers { permanent_uuid: "6a29d1c1c84644fea8558af4f2c9823f" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43897 } } peers { permanent_uuid: "667b4a82651c4a26a717f37627d060d3" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34699 } }
I20260925 19:05:18.688265 8242 leader_election.cc:288] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c [CANDIDATE]: Term 1 election: Requested vote from peers 6a29d1c1c84644fea8558af4f2c9823f (127.0.196.194:43897), 667b4a82651c4a26a717f37627d060d3 (127.0.196.193:34699)
I20260925 19:05:18.688405 7815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea0144686a2442ad83097220e178b7c4" candidate_uuid: "4e5e4690e8104ca09a7592a858d2983c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "667b4a82651c4a26a717f37627d060d3"
I20260925 19:05:18.688427 7950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea0144686a2442ad83097220e178b7c4" candidate_uuid: "4e5e4690e8104ca09a7592a858d2983c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a29d1c1c84644fea8558af4f2c9823f"
I20260925 19:05:18.688495 7950 raft_consensus.cc:3037] T ea0144686a2442ad83097220e178b7c4 P 6a29d1c1c84644fea8558af4f2c9823f [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:18.688503 7815 raft_consensus.cc:3037] T ea0144686a2442ad83097220e178b7c4 P 667b4a82651c4a26a717f37627d060d3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:18.689234 7950 raft_consensus.cc:2454] T ea0144686a2442ad83097220e178b7c4 P 6a29d1c1c84644fea8558af4f2c9823f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e5e4690e8104ca09a7592a858d2983c in term 1.
I20260925 19:05:18.689242 7815 raft_consensus.cc:2454] T ea0144686a2442ad83097220e178b7c4 P 667b4a82651c4a26a717f37627d060d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e5e4690e8104ca09a7592a858d2983c in term 1.
I20260925 19:05:18.689373 8021 leader_election.cc:302] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e5e4690e8104ca09a7592a858d2983c, 6a29d1c1c84644fea8558af4f2c9823f; no voters:
I20260925 19:05:18.689466 8242 raft_consensus.cc:2781] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:05:18.689623 8242 raft_consensus.cc:686] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c [term 1 LEADER]: Becoming Leader. State: Replica: 4e5e4690e8104ca09a7592a858d2983c, State: Running, Role: LEADER
I20260925 19:05:18.689821 8242 consensus_queue.cc:237] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5e4690e8104ca09a7592a858d2983c" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 36681 } } peers { permanent_uuid: "6a29d1c1c84644fea8558af4f2c9823f" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43897 } } peers { permanent_uuid: "667b4a82651c4a26a717f37627d060d3" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34699 } }
I20260925 19:05:18.689966 8215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.689849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.196.195:46973)
0925 19:05:18.689903 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:18.689940 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 19:05:18.690553 7493 catalog_manager.cc:5495] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e5e4690e8104ca09a7592a858d2983c (127.0.196.195). New cstate: current_term: 1 leader_uuid: "4e5e4690e8104ca09a7592a858d2983c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e5e4690e8104ca09a7592a858d2983c" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a29d1c1c84644fea8558af4f2c9823f" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "667b4a82651c4a26a717f37627d060d3" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34699 } health_report { overall_health: UNKNOWN } } }
I20260925 19:05:18.698838 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.696331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45350 (local address 127.0.196.195:36681)
0925 19:05:18.696383 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:18.696386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:18.696392 (+ 6us) server_negotiation.cc:407] Connection header received
0925 19:05:18.696433 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:18.696434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:18.696463 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:18.696551 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:05:18.696678 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:18.697165 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:18.697975 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:18.698185 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:18.698284 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:18.698456 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:05:18.698566 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:05:18.698669 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:18.698732 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 19:05:18.698753 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:05:18.699993 7950 raft_consensus.cc:1261] T ea0144686a2442ad83097220e178b7c4 P 6a29d1c1c84644fea8558af4f2c9823f [term 1 FOLLOWER]: Refusing update from remote peer 4e5e4690e8104ca09a7592a858d2983c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:05:18.700129 7815 raft_consensus.cc:1261] T ea0144686a2442ad83097220e178b7c4 P 667b4a82651c4a26a717f37627d060d3 [term 1 FOLLOWER]: Refusing update from remote peer 4e5e4690e8104ca09a7592a858d2983c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:05:18.700268 8242 consensus_queue.cc:1035] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a29d1c1c84644fea8558af4f2c9823f" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:05:18.700409 8243 consensus_queue.cc:1035] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c [LEADER]: Connected to new peer: Peer: permanent_uuid: "667b4a82651c4a26a717f37627d060d3" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:05:18.701447 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.701196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.196.194:54639)
0925 19:05:18.701365 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:18.701410 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260925 19:05:18.701592 8214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.701317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.196.193:49181)
0925 19:05:18.701535 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:18.701565 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203}
I20260925 19:05:18.702538 8215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.702447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.196.195:47473)
0925 19:05:18.702480 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:18.702505 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260925 19:05:18.702875 8215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.702786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.196.195:40549)
0925 19:05:18.702816 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:18.702843 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260925 19:05:18.703310 8214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.703210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.196.193:60761)
0925 19:05:18.703246 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:18.703270 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 19:05:18.703991 8247 mvcc.cc:205] Tried to move back new op lower bound from 7333327334193045504 to 7333327334154125312. Current Snapshot: MvccSnapshot[applied={T|T < 7333327334193045504 or (T in {7333327334193045504})}]
I20260925 19:05:18.704269 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.704167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38655 (local address 127.0.196.194:43535)
0925 19:05:18.704209 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:18.704242 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 19:05:18.705170 7493 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:05:18.705272 7493 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:05:18.706460 7493 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc40919e46b24cbebc864883e6aa8e29: Sending DeleteTablet for 3 replicas of tablet ea0144686a2442ad83097220e178b7c4
I20260925 19:05:18.706734 8067 tablet_service.cc:1514] Processing DeleteTablet for tablet ea0144686a2442ad83097220e178b7c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:05:18 UTC) from {username='oryx', principal='oryx/127.0.196.253@KRBTEST.COM'} at 127.0.0.1:45334
I20260925 19:05:18.706844 7931 tablet_service.cc:1514] Processing DeleteTablet for tablet ea0144686a2442ad83097220e178b7c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:05:18 UTC) from {username='oryx', principal='oryx/127.0.196.253@KRBTEST.COM'} at 127.0.0.1:59916
I20260925 19:05:18.706933 8256 tablet_replica.cc:330] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c: stopping tablet replica
I20260925 19:05:18.706990 8256 raft_consensus.cc:2229] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:05:18.707012 8257 tablet_replica.cc:330] T ea0144686a2442ad83097220e178b7c4 P 6a29d1c1c84644fea8558af4f2c9823f: stopping tablet replica
I20260925 19:05:18.707054 8256 raft_consensus.cc:2258] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:05:18.707062 8257 raft_consensus.cc:2229] T ea0144686a2442ad83097220e178b7c4 P 6a29d1c1c84644fea8558af4f2c9823f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:05:18.707057 7794 tablet_service.cc:1514] Processing DeleteTablet for tablet ea0144686a2442ad83097220e178b7c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:05:18 UTC) from {username='oryx', principal='oryx/127.0.196.253@KRBTEST.COM'} at 127.0.0.1:32946
I20260925 19:05:18.707104 8257 raft_consensus.cc:2258] T ea0144686a2442ad83097220e178b7c4 P 6a29d1c1c84644fea8558af4f2c9823f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:05:18.707408 8257 ts_tablet_manager.cc:1905] T ea0144686a2442ad83097220e178b7c4 P 6a29d1c1c84644fea8558af4f2c9823f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:05:18.707679 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 7732
I20260925 19:05:18.709942 8257 ts_tablet_manager.cc:1918] T ea0144686a2442ad83097220e178b7c4 P 6a29d1c1c84644fea8558af4f2c9823f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:05:18.710009 8257 log.cc:1196] T ea0144686a2442ad83097220e178b7c4 P 6a29d1c1c84644fea8558af4f2c9823f: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-1/wal/wals/ea0144686a2442ad83097220e178b7c4
I20260925 19:05:18.710234 8257 ts_tablet_manager.cc:1939] T ea0144686a2442ad83097220e178b7c4 P 6a29d1c1c84644fea8558af4f2c9823f: Deleting consensus metadata
I20260925 19:05:18.710669 7479 catalog_manager.cc:4841] TS 6a29d1c1c84644fea8558af4f2c9823f (127.0.196.194:43897): tablet ea0144686a2442ad83097220e178b7c4 (table test-table [id=0e1e5d1efa7f4281b143df7a098dcece]) successfully deleted
I20260925 19:05:18.715075 8256 ts_tablet_manager.cc:1905] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:05:18.716749 8256 ts_tablet_manager.cc:1918] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:05:18.716888 8256 log.cc:1196] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790363087086945-787-0/minicluster-data/ts-2/wal/wals/ea0144686a2442ad83097220e178b7c4
I20260925 19:05:18.717041 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 7868
I20260925 19:05:18.717597 8229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:18.717093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:34699 (local address 127.0.0.1:32962)
0925 19:05:18.717527 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:18.717562 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.193:34699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400}
W20260925 19:05:18.717768 7481 catalog_manager.cc:4584] TS 667b4a82651c4a26a717f37627d060d3 (127.0.196.193:34699): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea0144686a2442ad83097220e178b7c4: Network error: Client connection negotiation failed: client connection to 127.0.196.193:34699: connect: Connection refused (error 111)
I20260925 19:05:18.718075 8256 ts_tablet_manager.cc:1939] T ea0144686a2442ad83097220e178b7c4 P 4e5e4690e8104ca09a7592a858d2983c: Deleting consensus metadata
I20260925 19:05:18.718367 7478 catalog_manager.cc:4841] TS 4e5e4690e8104ca09a7592a858d2983c (127.0.196.195:36681): tablet ea0144686a2442ad83097220e178b7c4 (table test-table [id=0e1e5d1efa7f4281b143df7a098dcece]) successfully deleted
I20260925 19:05:18.722023 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 8004
I20260925 19:05:18.726475 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 7360
I20260925 19:05:18.731249 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 7548
2026-09-25T19:05:18Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6557 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:05:18 dist-test-slave-rn98 krb5kdc[8261](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:05:18 dist-test-slave-rn98 krb5kdc[8261](info): set up 2 sockets
Sep 25 19:05:18 dist-test-slave-rn98 krb5kdc[8261](info): commencing operation
krb5kdc: starting...
W20260925 19:05:19.779256 787 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:05:19 dist-test-slave-rn98 krb5kdc[8261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:05:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:05:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/oryx_127.0.196.254.keytab.
Entry for principal oryx/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/oryx_127.0.196.254.keytab.
I20260925 19:05:19.905910 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:41331
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/oryx_127.0.196.254.keytab
--principal=oryx/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:34171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:41331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:19.987504 8274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:19.987737 8274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:19.987761 8274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:19.988999 8274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:05:19.989054 8274 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 19:05:19.989068 8274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:19.989082 8274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:19.989138 8274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:05:19.989154 8274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:05:19.990723 8274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:34171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:41331
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/oryx_127.0.196.254.keytab
--principal=oryx/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:41331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:19.991022 8274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:19.991276 8274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:19.996388 8279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:19.996536 8274 server_base.cc:1034] running on GCE node
W20260925 19:05:19.996398 8282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:19.996423 8280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:19.996917 8274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:19.997205 8274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:19.998358 8274 hybrid_clock.cc:648] HybridClock initialized: now 1790363119998337 us; error 39 us; skew 500 ppm
Sep 25 19:05:19 dist-test-slave-rn98 krb5kdc[8261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363119, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:20.001166 8274 init.cc:377] Logged in from keytab as oryx/127.0.196.254@KRBTEST.COM (short username oryx)
I20260925 19:05:20.001693 8274 webserver.cc:460] Webserver started at http://127.0.196.254:41099/ using document root <none> and password file <none>
I20260925 19:05:20.001965 8274 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:20.002041 8274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:20.002202 8274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:20.003422 8274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "629f983bc1f14546b301430b0bbe9824"
format_stamp: "Formatted at 2026-09-25 19:05:19 on dist-test-slave-rn98"
server_key: "95c4080bc02a85ce70e402491be871c0"
server_key_iv: "8f865632eec6b3491372145f845f94ae"
server_key_version: "encryptionkey@0"
I20260925 19:05:20.003813 8274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "629f983bc1f14546b301430b0bbe9824"
format_stamp: "Formatted at 2026-09-25 19:05:19 on dist-test-slave-rn98"
server_key: "95c4080bc02a85ce70e402491be871c0"
server_key_iv: "8f865632eec6b3491372145f845f94ae"
server_key_version: "encryptionkey@0"
I20260925 19:05:20.005249 8274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 19:05:20.006129 8289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:20.006333 8274 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:20.006453 8274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "629f983bc1f14546b301430b0bbe9824"
format_stamp: "Formatted at 2026-09-25 19:05:19 on dist-test-slave-rn98"
server_key: "95c4080bc02a85ce70e402491be871c0"
server_key_iv: "8f865632eec6b3491372145f845f94ae"
server_key_version: "encryptionkey@0"
I20260925 19:05:20.006557 8274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:20.018060 8274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:20.020543 8274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:20.020740 8274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:20.025615 8274 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:41331
I20260925 19:05:20.025686 8351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:41331 every 8 connection(s)
I20260925 19:05:20.026068 8274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:05:20.026763 8352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:20.029194 8352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 629f983bc1f14546b301430b0bbe9824: Bootstrap starting.
I20260925 19:05:20.029832 8352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 629f983bc1f14546b301430b0bbe9824: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:20.030190 8352 log.cc:824] T 00000000000000000000000000000000 P 629f983bc1f14546b301430b0bbe9824: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:20.031131 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 8274
I20260925 19:05:20.031226 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:05:20.031224 8352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 629f983bc1f14546b301430b0bbe9824: No bootstrap required, opened a new log
I20260925 19:05:20.031430 787 external_mini_cluster.cc:1428] Setting key bfee2221ea00afe45ace286331c25bea
I20260925 19:05:20.034243 8352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 629f983bc1f14546b301430b0bbe9824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "629f983bc1f14546b301430b0bbe9824" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41331 } }
I20260925 19:05:20.034440 8352 raft_consensus.cc:374] T 00000000000000000000000000000000 P 629f983bc1f14546b301430b0bbe9824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:20.034492 8352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 629f983bc1f14546b301430b0bbe9824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 629f983bc1f14546b301430b0bbe9824, State: Initialized, Role: FOLLOWER
I20260925 19:05:20.034675 8352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 629f983bc1f14546b301430b0bbe9824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "629f983bc1f14546b301430b0bbe9824" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41331 } }
I20260925 19:05:20.034771 8352 raft_consensus.cc:388] T 00000000000000000000000000000000 P 629f983bc1f14546b301430b0bbe9824 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:05:20.034816 8352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 629f983bc1f14546b301430b0bbe9824 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:05:20.034873 8352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 629f983bc1f14546b301430b0bbe9824 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:20.035650 8352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 629f983bc1f14546b301430b0bbe9824 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "629f983bc1f14546b301430b0bbe9824" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41331 } }
I20260925 19:05:20.035761 8352 leader_election.cc:302] T 00000000000000000000000000000000 P 629f983bc1f14546b301430b0bbe9824 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 629f983bc1f14546b301430b0bbe9824; no voters:
I20260925 19:05:20.035938 8352 leader_election.cc:288] T 00000000000000000000000000000000 P 629f983bc1f14546b301430b0bbe9824 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:05:20.036053 8357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 629f983bc1f14546b301430b0bbe9824 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:05:20.036211 8352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 629f983bc1f14546b301430b0bbe9824 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:05:20.036309 8357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 629f983bc1f14546b301430b0bbe9824 [term 1 LEADER]: Becoming Leader. State: Replica: 629f983bc1f14546b301430b0bbe9824, State: Running, Role: LEADER
I20260925 19:05:20.036454 8357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 629f983bc1f14546b301430b0bbe9824 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "629f983bc1f14546b301430b0bbe9824" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41331 } }
I20260925 19:05:20.036914 8358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 629f983bc1f14546b301430b0bbe9824 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "629f983bc1f14546b301430b0bbe9824" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "629f983bc1f14546b301430b0bbe9824" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41331 } } }
I20260925 19:05:20.036996 8358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 629f983bc1f14546b301430b0bbe9824 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:20.037145 8359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 629f983bc1f14546b301430b0bbe9824 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 629f983bc1f14546b301430b0bbe9824. Latest consensus state: current_term: 1 leader_uuid: "629f983bc1f14546b301430b0bbe9824" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "629f983bc1f14546b301430b0bbe9824" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41331 } } }
I20260925 19:05:20.037204 8359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 629f983bc1f14546b301430b0bbe9824 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:20.037634 8366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:05:20.038213 8366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 19:05:20 dist-test-slave-rn98 krb5kdc[8261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM
I20260925 19:05:20.040207 8366 catalog_manager.cc:1347] Generated new cluster ID: b6c36cda42f84808a25664b84075b3ec
I20260925 19:05:20.040264 8366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:05:20.044493 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.032023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43704 (local address 127.0.196.254:41331)
0925 19:05:20.032219 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:20.032228 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:20.032254 (+ 26us) server_negotiation.cc:407] Connection header received
0925 19:05:20.032678 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:20.032685 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:20.032734 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:20.032857 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:20.033868 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.035052 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.036165 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.036830 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.036941 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.040425 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:20.040430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:20.040430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:20.040445 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:20.042572 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:20.043037 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:20.043041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:20.043041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:20.043166 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:20.043381 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:20.043382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:20.043383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:20.043741 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:20.043972 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:20.044079 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 19:05:20.044243 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":107,"threads_started":1}
I20260925 19:05:20.051298 8366 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:05:20.051890 8366 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:05:20.054211 8366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 629f983bc1f14546b301430b0bbe9824: Generated new TSK 0
I20260925 19:05:20.054438 8366 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/oryx_127.0.196.193.keytab.
Entry for principal oryx/127.0.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/oryx_127.0.196.193.keytab.
I20260925 19:05:20.087815 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:0
--local_ip_for_outbound_sockets=127.0.196.193
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:41331
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/oryx_127.0.196.193.keytab
--principal=oryx/127.0.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:34171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:20.173184 8378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:20.173460 8378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:20.173507 8378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:20.174885 8378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:20.174958 8378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:20.174995 8378 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 19:05:20.175052 8378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:05:20.176790 8378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:34171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/oryx_127.0.196.193.keytab
--principal=oryx/127.0.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.196.254:41331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:20.177099 8378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:20.177356 8378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:20.182701 8384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:20.182713 8386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:20.182722 8383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:20.183084 8378 server_base.cc:1034] running on GCE node
I20260925 19:05:20.183514 8378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:20.183846 8378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:20.185057 8378 hybrid_clock.cc:648] HybridClock initialized: now 1790363120185036 us; error 42 us; skew 500 ppm
Sep 25 19:05:20 dist-test-slave-rn98 krb5kdc[8261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363120, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:20.187860 8378 init.cc:377] Logged in from keytab as oryx/127.0.196.193@KRBTEST.COM (short username oryx)
I20260925 19:05:20.188361 8378 webserver.cc:460] Webserver started at http://127.0.196.193:46815/ using document root <none> and password file <none>
I20260925 19:05:20.188598 8378 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:20.188666 8378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:20.188827 8378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:20.189898 8378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-0/data/instance:
uuid: "df934d2b8daf44219344ce39308d83e5"
format_stamp: "Formatted at 2026-09-25 19:05:20 on dist-test-slave-rn98"
server_key: "94c6b43fa3df3bdefdae1b30e5befbfa"
server_key_iv: "08ccd0254e832277184d70864de776ac"
server_key_version: "encryptionkey@0"
I20260925 19:05:20.190284 8378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-0/wal/instance:
uuid: "df934d2b8daf44219344ce39308d83e5"
format_stamp: "Formatted at 2026-09-25 19:05:20 on dist-test-slave-rn98"
server_key: "94c6b43fa3df3bdefdae1b30e5befbfa"
server_key_iv: "08ccd0254e832277184d70864de776ac"
server_key_version: "encryptionkey@0"
I20260925 19:05:20.191743 8378 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:05:20.192667 8393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:20.192879 8378 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:05:20.193053 8378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "df934d2b8daf44219344ce39308d83e5"
format_stamp: "Formatted at 2026-09-25 19:05:20 on dist-test-slave-rn98"
server_key: "94c6b43fa3df3bdefdae1b30e5befbfa"
server_key_iv: "08ccd0254e832277184d70864de776ac"
server_key_version: "encryptionkey@0"
I20260925 19:05:20.193120 8378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:20.201930 8378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:20.204351 8378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:20.204474 8378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:20.205053 8378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:20.205149 8378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:20.205207 8378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:20.205238 8378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 19:05:20 dist-test-slave-rn98 krb5kdc[8261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363120, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM
I20260925 19:05:20.212931 8378 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:46635
I20260925 19:05:20.213313 8378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
I20260925 19:05:20.213773 8508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:46635 every 8 connection(s)
I20260925 19:05:20.216445 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.205500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:52931 (local address 127.0.196.254:41331)
0925 19:05:20.205572 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:20.205575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:20.205869 (+ 294us) server_negotiation.cc:407] Connection header received
0925 19:05:20.206662 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:20.206664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:20.206678 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:20.206742 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:20.207776 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.208291 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.210158 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.210505 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.210581 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.213479 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:20.213481 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:20.213482 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:20.213491 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:20.215131 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:20.215549 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:20.215550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:20.215550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:20.215604 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:20.215816 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:20.215816 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:20.215816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:20.215934 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:20.215984 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:20.216293 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 19:05:20.216370 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 19:05:20.216527 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.205497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41331 (local address 127.0.196.193:52931)
0925 19:05:20.205699 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:20.205756 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:20.206495 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:20.206848 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:20.206853 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:20.207277 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:20.207654 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.207663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.208369 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.208370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.210025 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.210030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.210602 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.210603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.210823 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.211423 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:20.211435 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:20.213359 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:20.215239 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:20.215245 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:20.215247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:20.215474 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:20.215663 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:20.215663 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:20.215664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:20.215761 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:20.216026 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:20.216030 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:20.216147 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 19:05:20.216235 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 19:05:20.216363 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":102,"threads_started":1}
I20260925 19:05:20.217047 8509 heartbeater.cc:347] Connected to a master server at 127.0.196.254:41331
I20260925 19:05:20.217129 8509 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:20.217329 8509 heartbeater.cc:511] Master 127.0.196.254:41331 requested a full tablet report, sending...
I20260925 19:05:20.217979 8305 ts_manager.cc:194] Registered new tserver with Master: df934d2b8daf44219344ce39308d83e5 (127.0.196.193:46635)
I20260925 19:05:20.218963 8305 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.196.193@KRBTEST.COM'} at 127.0.196.193:52931
I20260925 19:05:20.223519 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 8378
I20260925 19:05:20.223623 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-0/wal/instance
I20260925 19:05:20.223843 787 external_mini_cluster.cc:1428] Setting key beec9e1589f511f4d784311acf94d1d0
I20260925 19:05:20.224054 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.218121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41331 (local address 127.0.196.193:33779)
0925 19:05:20.218259 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:20.218269 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:20.218327 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:20.218450 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:20.218451 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:20.218666 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:20.218736 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.218738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.219496 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.219497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.220093 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.220094 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.220423 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.220424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.220476 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.220900 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:20.220906 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:20.221197 (+ 291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:20.222783 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:20.222784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:20.222784 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:20.222999 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:20.223286 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:20.223287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:20.223287 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:20.223355 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:20.223689 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:20.223690 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:20.223747 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 19:05:20.223827 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 19:05:20.223943 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":24,"thread_start_us":77,"threads_started":1}
I20260925 19:05:20.224143 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.218160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:33779 (local address 127.0.196.254:41331)
0925 19:05:20.218195 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:20.218197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:20.218311 (+ 114us) server_negotiation.cc:407] Connection header received
0925 19:05:20.218391 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:20.218392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:20.218400 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:20.218445 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:20.218803 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.219408 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.220169 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.220350 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.220408 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.221278 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:20.221279 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:20.221279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:20.221285 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:20.222704 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:20.223075 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:20.223075 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:20.223075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:20.223127 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:20.223431 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:20.223432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:20.223433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:20.223580 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:20.223656 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:20.223869 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 19:05:20.224035 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
WARNING: no policy specified for oryx/127.0.196.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/oryx_127.0.196.194.keytab.
Entry for principal oryx/127.0.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/oryx_127.0.196.194.keytab.
I20260925 19:05:20.250514 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:0
--local_ip_for_outbound_sockets=127.0.196.194
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:41331
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/oryx_127.0.196.194.keytab
--principal=oryx/127.0.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:34171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:20.333699 8518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:20.333977 8518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:20.334019 8518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:20.335242 8518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:20.335317 8518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:20.335353 8518 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 19:05:20.335392 8518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:05:20.337069 8518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:34171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/oryx_127.0.196.194.keytab
--principal=oryx/127.0.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.196.254:41331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:20.337364 8518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:20.337646 8518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:20.342903 8526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:20.343086 8524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:20.342903 8523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:20.343253 8518 server_base.cc:1034] running on GCE node
I20260925 19:05:20.343554 8518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:20.343796 8518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:20.344959 8518 hybrid_clock.cc:648] HybridClock initialized: now 1790363120344942 us; error 33 us; skew 500 ppm
Sep 25 19:05:20 dist-test-slave-rn98 krb5kdc[8261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363120, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:20.347819 8518 init.cc:377] Logged in from keytab as oryx/127.0.196.194@KRBTEST.COM (short username oryx)
I20260925 19:05:20.348369 8518 webserver.cc:460] Webserver started at http://127.0.196.194:41875/ using document root <none> and password file <none>
I20260925 19:05:20.348639 8518 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:20.348709 8518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:20.348865 8518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:20.349973 8518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-1/data/instance:
uuid: "b81a2855b68448ad9c7d40da4be26ecd"
format_stamp: "Formatted at 2026-09-25 19:05:20 on dist-test-slave-rn98"
server_key: "df35ddecc047f0c79883b5a208c0c319"
server_key_iv: "a9362d299880374d46cf1bf8eea3b927"
server_key_version: "encryptionkey@0"
I20260925 19:05:20.350351 8518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-1/wal/instance:
uuid: "b81a2855b68448ad9c7d40da4be26ecd"
format_stamp: "Formatted at 2026-09-25 19:05:20 on dist-test-slave-rn98"
server_key: "df35ddecc047f0c79883b5a208c0c319"
server_key_iv: "a9362d299880374d46cf1bf8eea3b927"
server_key_version: "encryptionkey@0"
I20260925 19:05:20.351881 8518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:05:20.352727 8533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:20.352922 8518 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:20.353011 8518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "b81a2855b68448ad9c7d40da4be26ecd"
format_stamp: "Formatted at 2026-09-25 19:05:20 on dist-test-slave-rn98"
server_key: "df35ddecc047f0c79883b5a208c0c319"
server_key_iv: "a9362d299880374d46cf1bf8eea3b927"
server_key_version: "encryptionkey@0"
I20260925 19:05:20.353132 8518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:20.366160 8518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:20.368775 8518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:20.368983 8518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:20.369675 8518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:20.369730 8518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:20.369773 8518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:20.369817 8518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 19:05:20 dist-test-slave-rn98 krb5kdc[8261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363120, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM
I20260925 19:05:20.377462 8518 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:34129
I20260925 19:05:20.377873 8518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
I20260925 19:05:20.377840 8648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:34129 every 8 connection(s)
I20260925 19:05:20.380589 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.370131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:39803 (local address 127.0.196.254:41331)
0925 19:05:20.370201 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:20.370204 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:20.370479 (+ 275us) server_negotiation.cc:407] Connection header received
0925 19:05:20.371234 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:20.371235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:20.371249 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:20.371296 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:20.372071 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.372628 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.373766 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.374005 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.374102 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.377153 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:20.377155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:20.377155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:20.377171 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:20.379023 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:20.379490 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:20.379492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:20.379493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:20.379562 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:20.379852 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:20.379853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:20.379854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:20.379997 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:20.380081 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:20.380388 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 19:05:20.380479 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 19:05:20.380822 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.370133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41331 (local address 127.0.196.194:39803)
0925 19:05:20.370320 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:20.370366 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:20.371103 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:20.371388 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:20.371394 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:20.371706 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:20.371938 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.371944 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.372746 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.372747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.373662 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.373664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.374115 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.374116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.374335 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.375063 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:20.375076 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:20.377033 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:20.379129 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:20.379135 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:20.379138 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:20.379387 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:20.379640 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:20.379641 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:20.379641 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:20.379775 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:20.380072 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:20.380079 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:20.380239 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 19:05:20.380366 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 19:05:20.380558 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":114,"threads_started":1}
I20260925 19:05:20.381435 8649 heartbeater.cc:347] Connected to a master server at 127.0.196.254:41331
I20260925 19:05:20.381531 8649 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:20.381733 8649 heartbeater.cc:511] Master 127.0.196.254:41331 requested a full tablet report, sending...
I20260925 19:05:20.382141 8305 ts_manager.cc:194] Registered new tserver with Master: b81a2855b68448ad9c7d40da4be26ecd (127.0.196.194:34129)
I20260925 19:05:20.382539 8305 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.196.194@KRBTEST.COM'} at 127.0.196.194:39803
I20260925 19:05:20.385879 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 8518
I20260925 19:05:20.386013 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-1/wal/instance
I20260925 19:05:20.386250 787 external_mini_cluster.cc:1428] Setting key f51ff7c6ea6ddaedb2a99f8822eae933
I20260925 19:05:20.389115 8655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.382597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41331 (local address 127.0.196.194:34623)
0925 19:05:20.382768 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:20.382779 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:20.382848 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:20.383076 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:20.383077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:20.383313 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:20.383389 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.383391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.384061 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.384062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.384624 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.384625 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.385014 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.385015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.385069 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.385527 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:20.385533 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:20.385881 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:20.387817 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:20.387818 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:20.387819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:20.388106 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:20.388327 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:20.388328 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:20.388328 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:20.388414 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:20.388808 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:20.388808 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:20.388862 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 19:05:20.388923 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 19:05:20.389030 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":119,"threads_started":1}
I20260925 19:05:20.389258 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.382669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:34623 (local address 127.0.196.254:41331)
0925 19:05:20.382710 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:20.382712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:20.382855 (+ 143us) server_negotiation.cc:407] Connection header received
0925 19:05:20.382886 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:20.382886 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:20.382981 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:20.383040 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:20.383471 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.383965 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.384702 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.384880 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.384996 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.386021 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:20.386023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:20.386024 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:20.386032 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:20.387725 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:20.388194 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:20.388195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:20.388195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:20.388253 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:20.388492 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:20.388493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:20.388493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:20.388733 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:20.388809 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:20.389031 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 19:05:20.389139 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
WARNING: no policy specified for oryx/127.0.196.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/oryx_127.0.196.195.keytab.
Entry for principal oryx/127.0.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/oryx_127.0.196.195.keytab.
I20260925 19:05:20.414389 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:0
--local_ip_for_outbound_sockets=127.0.196.195
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:41331
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/oryx_127.0.196.195.keytab
--principal=oryx/127.0.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:34171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:20.505365 8658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:20.505625 8658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:20.505654 8658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:20.506978 8658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:20.507019 8658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:20.507042 8658 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 19:05:20.507061 8658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:05:20.508772 8658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:34171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/krb5kdc/oryx_127.0.196.195.keytab
--principal=oryx/127.0.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.196.254:41331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:20.509069 8658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:20.509280 8658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:20.514837 8666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:20.514822 8664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:20.514819 8663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:20.514894 8658 server_base.cc:1034] running on GCE node
I20260925 19:05:20.515288 8658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:20.515514 8658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:20.516661 8658 hybrid_clock.cc:648] HybridClock initialized: now 1790363120516652 us; error 30 us; skew 500 ppm
Sep 25 19:05:20 dist-test-slave-rn98 krb5kdc[8261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363120, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:20.519426 8658 init.cc:377] Logged in from keytab as oryx/127.0.196.195@KRBTEST.COM (short username oryx)
I20260925 19:05:20.519953 8658 webserver.cc:460] Webserver started at http://127.0.196.195:36031/ using document root <none> and password file <none>
I20260925 19:05:20.520205 8658 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:20.520273 8658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:20.520416 8658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:20.521493 8658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-2/data/instance:
uuid: "0743aa7c73d64ffc8659e88d576805c0"
format_stamp: "Formatted at 2026-09-25 19:05:20 on dist-test-slave-rn98"
server_key: "05d043aabd3725e8a56cfba1f960425e"
server_key_iv: "f3ff0f84916b6e1fbac999c6b8f4f8a7"
server_key_version: "encryptionkey@0"
I20260925 19:05:20.521899 8658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-2/wal/instance:
uuid: "0743aa7c73d64ffc8659e88d576805c0"
format_stamp: "Formatted at 2026-09-25 19:05:20 on dist-test-slave-rn98"
server_key: "05d043aabd3725e8a56cfba1f960425e"
server_key_iv: "f3ff0f84916b6e1fbac999c6b8f4f8a7"
server_key_version: "encryptionkey@0"
I20260925 19:05:20.523470 8658 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:05:20.524288 8673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:20.524497 8658 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:05:20.524602 8658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "0743aa7c73d64ffc8659e88d576805c0"
format_stamp: "Formatted at 2026-09-25 19:05:20 on dist-test-slave-rn98"
server_key: "05d043aabd3725e8a56cfba1f960425e"
server_key_iv: "f3ff0f84916b6e1fbac999c6b8f4f8a7"
server_key_version: "encryptionkey@0"
I20260925 19:05:20.524684 8658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:20.551414 8658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:20.553750 8658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:20.553912 8658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:20.554730 8658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:20.554800 8658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:20.554847 8658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:20.554868 8658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 19:05:20 dist-test-slave-rn98 krb5kdc[8261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363120, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM
I20260925 19:05:20.562647 8658 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:44785
I20260925 19:05:20.562808 8788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:44785 every 8 connection(s)
I20260925 19:05:20.563091 8658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
I20260925 19:05:20.566053 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.555123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:48807 (local address 127.0.196.254:41331)
0925 19:05:20.555193 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:20.555196 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:20.555490 (+ 294us) server_negotiation.cc:407] Connection header received
0925 19:05:20.556023 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:20.556025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:20.556041 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:20.556133 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:20.557168 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.557772 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.559037 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.559264 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.559370 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.562457 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:20.562459 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:20.562459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:20.562469 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:20.564413 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:20.564892 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:20.564893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:20.564894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:20.564964 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:20.565254 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:20.565255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:20.565256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:20.565396 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:20.565479 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:20.565796 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 19:05:20.565953 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 19:05:20.566323 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.555134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41331 (local address 127.0.196.195:48807)
0925 19:05:20.555341 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:20.555387 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:20.555923 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:20.556305 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:20.556309 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:20.556737 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:20.557050 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.557057 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.557897 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.557898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.558925 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.558928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.559396 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.559397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.559559 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.560270 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:20.560284 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:20.562317 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:20.564530 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:20.564537 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:20.564539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:20.564799 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:20.565048 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:20.565049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:20.565049 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:20.565177 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:20.565484 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:20.565491 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:20.565651 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 19:05:20.565774 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 19:05:20.565958 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":112,"threads_started":1}
I20260925 19:05:20.566918 8789 heartbeater.cc:347] Connected to a master server at 127.0.196.254:41331
I20260925 19:05:20.567010 8789 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:20.567221 8789 heartbeater.cc:511] Master 127.0.196.254:41331 requested a full tablet report, sending...
I20260925 19:05:20.567911 8306 ts_manager.cc:194] Registered new tserver with Master: 0743aa7c73d64ffc8659e88d576805c0 (127.0.196.195:44785)
I20260925 19:05:20.568326 8306 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.196.195@KRBTEST.COM'} at 127.0.196.195:48807
I20260925 19:05:20.570294 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 8658
I20260925 19:05:20.570423 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-2/wal/instance
I20260925 19:05:20.570703 787 external_mini_cluster.cc:1428] Setting key 2ffa6980971d0fc28f46d18bd34a6874
I20260925 19:05:20.571887 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:05:20.574967 8795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.567951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41331 (local address 127.0.196.195:41351)
0925 19:05:20.568505 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:20.568517 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:20.568573 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:20.568709 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:20.568709 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:20.568870 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:20.568933 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.568935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.569741 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.569742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.570262 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.570264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.570866 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.570867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.570959 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.571638 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:20.571647 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:20.572053 (+ 406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:20.573816 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:20.573818 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:20.573819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:20.574018 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:20.574244 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:20.574245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:20.574245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:20.574311 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:20.574584 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:20.574585 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:20.574656 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 19:05:20.574725 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 19:05:20.574843 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":81,"threads_started":1}
I20260925 19:05:20.575027 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.568038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:41351 (local address 127.0.196.254:41331)
0925 19:05:20.568080 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:20.568082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:20.568583 (+ 501us) server_negotiation.cc:407] Connection header received
0925 19:05:20.568618 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:20.568619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:20.568631 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:20.568707 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:20.569007 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.569640 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.570376 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.570746 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.570837 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.572145 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:20.572146 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:20.572147 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:20.572157 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:20.573680 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:20.574097 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:20.574098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:20.574099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:20.574158 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:20.574380 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:20.574381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:20.574382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:20.574506 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:20.574579 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:20.574760 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 19:05:20.574842 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 25 19:05:20 dist-test-slave-rn98 krb5kdc[8261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363120, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 19:05:20 dist-test-slave-rn98 krb5kdc[8261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:20 dist-test-slave-rn98 krb5kdc[8261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 19:05:20.595419 8356 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:20.589016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43714 (local address 127.0.196.254:41331)
0925 19:05:20.589085 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:20.589088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:20.589155 (+ 67us) server_negotiation.cc:407] Connection header received
0925 19:05:20.589259 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:20.589261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:20.589279 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:20.589351 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:20.590011 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.590579 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.591465 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.591677 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.591762 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.595294 (+ 3532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43714: BlockingRecv error: recv got EOF from 127.0.0.1:43714 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 19:05:20 dist-test-slave-rn98 krb5kdc[8261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363120, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM
I20260925 19:05:20.604823 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.596464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43724 (local address 127.0.196.254:41331)
0925 19:05:20.596591 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:20.596594 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:20.596676 (+ 82us) server_negotiation.cc:407] Connection header received
0925 19:05:20.596773 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:20.596774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:20.596785 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:20.596866 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:20.597448 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.597999 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.598903 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.599126 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.599222 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.601644 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:20.601645 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:20.601646 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:20.601659 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:20.603411 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:20.603884 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:20.603886 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:20.603887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:20.603954 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:20.604177 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:20.604179 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:20.604179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:20.604313 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:20.604450 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:20.604578 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 19:05:20.604675 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 19:05:20.606783 8306 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:05:20.608413 8306 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:05:20.615731 8811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.611216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:46635 (local address 127.0.0.1:38254)
0925 19:05:20.611357 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:20.611408 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:20.611482 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:20.612155 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:20.612155 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:20.612157 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:20.612213 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.612215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.613228 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.613229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.614680 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.614682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.615383 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.615384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.615475 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.615491 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 19:05:20.615563 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 19:05:20.615629 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":29,"thread_start_us":93,"threads_started":1}
I20260925 19:05:20.615809 8816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.611708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60802 (local address 127.0.196.195:44785)
0925 19:05:20.612114 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:20.612118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:20.612128 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:05:20.612164 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:20.612165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:20.612180 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:20.612258 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:20.612366 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.613415 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.614590 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.615425 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.615501 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.615576 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:20.615646 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 19:05:20.615688 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":97,"threads_started":1}
I20260925 19:05:20.615962 8814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.611453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36914 (local address 127.0.196.194:34129)
0925 19:05:20.611782 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:20.611789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:20.611800 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:05:20.611845 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:20.611847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:20.611863 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:20.611991 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:20.612173 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.613325 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.614709 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.615435 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.615501 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.615576 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:20.615787 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 19:05:20.615833 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":86,"threads_started":1}
I20260925 19:05:20.615962 8813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.611399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38254 (local address 127.0.196.193:46635)
0925 19:05:20.611934 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:20.611940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:20.611949 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:05:20.611990 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:20.611992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:20.612008 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:20.612085 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:20.612312 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.613130 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.614750 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.615278 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.615372 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.615469 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:20.615779 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 19:05:20.615827 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":96,"threads_started":1}
I20260925 19:05:20.615731 8812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.611465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:44785 (local address 127.0.0.1:60802)
0925 19:05:20.611772 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:20.611781 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:20.611815 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:20.612269 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:20.612270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:20.612272 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:20.612305 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.612306 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.613483 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.613484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.614497 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.614500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.615520 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.615520 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.615580 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.615582 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:05:20.615634 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 19:05:20.615667 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":87,"threads_started":1}
I20260925 19:05:20.616189 8815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.611248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:34129 (local address 127.0.0.1:36914)
0925 19:05:20.611610 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:20.611622 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:20.611676 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:20.612000 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:20.612003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:20.612005 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:20.612080 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.612082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.613430 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.613431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.614603 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.614606 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.615615 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.615616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.615689 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.615693 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:05:20.616044 (+ 351us) client_negotiation.cc:240] Negotiation successful
0925 19:05:20.616113 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":94,"threads_started":1}
I20260925 19:05:20.617175 8583 tablet_service.cc:1467] Processing CreateTablet for tablet 44f3a3501ca7406691cc37097a237ad2 (DEFAULT_TABLE table=test-table [id=25f6b007db9e49ef8acc0b8f754b83c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:05:20.617142 8443 tablet_service.cc:1467] Processing CreateTablet for tablet 44f3a3501ca7406691cc37097a237ad2 (DEFAULT_TABLE table=test-table [id=25f6b007db9e49ef8acc0b8f754b83c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:05:20.617166 8723 tablet_service.cc:1467] Processing CreateTablet for tablet 44f3a3501ca7406691cc37097a237ad2 (DEFAULT_TABLE table=test-table [id=25f6b007db9e49ef8acc0b8f754b83c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:05:20.617538 8583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f3a3501ca7406691cc37097a237ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:20.617540 8723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f3a3501ca7406691cc37097a237ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:20.617540 8443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f3a3501ca7406691cc37097a237ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:20.620820 8817 tablet_bootstrap.cc:492] T 44f3a3501ca7406691cc37097a237ad2 P b81a2855b68448ad9c7d40da4be26ecd: Bootstrap starting.
I20260925 19:05:20.621529 8818 tablet_bootstrap.cc:492] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0: Bootstrap starting.
I20260925 19:05:20.621879 8817 tablet_bootstrap.cc:654] T 44f3a3501ca7406691cc37097a237ad2 P b81a2855b68448ad9c7d40da4be26ecd: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:20.622206 8818 tablet_bootstrap.cc:654] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:20.622252 8817 log.cc:824] T 44f3a3501ca7406691cc37097a237ad2 P b81a2855b68448ad9c7d40da4be26ecd: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:20.622568 8818 log.cc:824] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:20.622656 8819 tablet_bootstrap.cc:492] T 44f3a3501ca7406691cc37097a237ad2 P df934d2b8daf44219344ce39308d83e5: Bootstrap starting.
I20260925 19:05:20.623144 8817 tablet_bootstrap.cc:492] T 44f3a3501ca7406691cc37097a237ad2 P b81a2855b68448ad9c7d40da4be26ecd: No bootstrap required, opened a new log
I20260925 19:05:20.623221 8817 ts_tablet_manager.cc:1397] T 44f3a3501ca7406691cc37097a237ad2 P b81a2855b68448ad9c7d40da4be26ecd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:05:20.623304 8818 tablet_bootstrap.cc:492] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0: No bootstrap required, opened a new log
I20260925 19:05:20.623368 8818 ts_tablet_manager.cc:1397] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:05:20.625355 8818 raft_consensus.cc:348] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.625355 8817 raft_consensus.cc:348] T 44f3a3501ca7406691cc37097a237ad2 P b81a2855b68448ad9c7d40da4be26ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.625550 8817 raft_consensus.cc:374] T 44f3a3501ca7406691cc37097a237ad2 P b81a2855b68448ad9c7d40da4be26ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:20.625550 8818 raft_consensus.cc:374] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:20.625586 8818 raft_consensus.cc:729] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0743aa7c73d64ffc8659e88d576805c0, State: Initialized, Role: FOLLOWER
I20260925 19:05:20.625586 8817 raft_consensus.cc:729] T 44f3a3501ca7406691cc37097a237ad2 P b81a2855b68448ad9c7d40da4be26ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81a2855b68448ad9c7d40da4be26ecd, State: Initialized, Role: FOLLOWER
I20260925 19:05:20.625739 8818 consensus_queue.cc:260] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.625769 8817 consensus_queue.cc:260] T 44f3a3501ca7406691cc37097a237ad2 P b81a2855b68448ad9c7d40da4be26ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.626050 8817 ts_tablet_manager.cc:1428] T 44f3a3501ca7406691cc37097a237ad2 P b81a2855b68448ad9c7d40da4be26ecd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:05:20.626051 8818 ts_tablet_manager.cc:1428] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:05:20.626181 8789 heartbeater.cc:503] Master 127.0.196.254:41331 was elected leader, sending a full tablet report...
I20260925 19:05:20.626183 8649 heartbeater.cc:503] Master 127.0.196.254:41331 was elected leader, sending a full tablet report...
I20260925 19:05:20.627411 8819 tablet_bootstrap.cc:654] T 44f3a3501ca7406691cc37097a237ad2 P df934d2b8daf44219344ce39308d83e5: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:20.627785 8819 log.cc:824] T 44f3a3501ca7406691cc37097a237ad2 P df934d2b8daf44219344ce39308d83e5: Log is configured to *not* fsync() on all Append() calls
W20260925 19:05:20.628779 8650 tablet.cc:2369] T 44f3a3501ca7406691cc37097a237ad2 P b81a2855b68448ad9c7d40da4be26ecd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:05:20.628990 8819 tablet_bootstrap.cc:492] T 44f3a3501ca7406691cc37097a237ad2 P df934d2b8daf44219344ce39308d83e5: No bootstrap required, opened a new log
I20260925 19:05:20.629078 8819 ts_tablet_manager.cc:1397] T 44f3a3501ca7406691cc37097a237ad2 P df934d2b8daf44219344ce39308d83e5: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260925 19:05:20.630810 8819 raft_consensus.cc:348] T 44f3a3501ca7406691cc37097a237ad2 P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.630971 8819 raft_consensus.cc:374] T 44f3a3501ca7406691cc37097a237ad2 P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:20.630996 8819 raft_consensus.cc:729] T 44f3a3501ca7406691cc37097a237ad2 P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df934d2b8daf44219344ce39308d83e5, State: Initialized, Role: FOLLOWER
I20260925 19:05:20.631126 8819 consensus_queue.cc:260] T 44f3a3501ca7406691cc37097a237ad2 P df934d2b8daf44219344ce39308d83e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.631412 8819 ts_tablet_manager.cc:1428] T 44f3a3501ca7406691cc37097a237ad2 P df934d2b8daf44219344ce39308d83e5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 19:05:20.631456 8509 heartbeater.cc:503] Master 127.0.196.254:41331 was elected leader, sending a full tablet report...
I20260925 19:05:20.712245 8823 raft_consensus.cc:482] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:05:20.712363 8823 raft_consensus.cc:504] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.712803 8823 leader_election.cc:288] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df934d2b8daf44219344ce39308d83e5 (127.0.196.193:46635), b81a2855b68448ad9c7d40da4be26ecd (127.0.196.194:34129)
W20260925 19:05:20.714097 8510 tablet.cc:2369] T 44f3a3501ca7406691cc37097a237ad2 P df934d2b8daf44219344ce39308d83e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:05:20.716040 8814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.713079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:41639 (local address 127.0.196.194:34129)
0925 19:05:20.713123 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:20.713125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:20.713135 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:05:20.713196 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:20.713197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:20.713206 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:20.713259 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:20.713461 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.714011 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.715189 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.715603 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.715714 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.715724 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:20.715913 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 19:05:20.715957 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:05:20.716058 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.713011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:34129 (local address 127.0.196.195:41639)
0925 19:05:20.713071 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:20.713084 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:20.713134 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:20.713264 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:20.713264 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:20.713265 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:20.713357 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.713358 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.714112 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.714113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.715079 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.715082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.715712 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.715713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.715817 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.715820 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:05:20.715908 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 19:05:20.715960 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 19:05:20.716077 8813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.713113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:39845 (local address 127.0.196.193:46635)
0925 19:05:20.713331 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:20.713334 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:20.713343 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:05:20.713374 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:20.713375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:20.713384 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:20.713489 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:20.713661 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.714285 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.715286 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.715691 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.715788 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.715796 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:20.715968 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 19:05:20.716006 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260925 19:05:20.716341 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.713038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:46635 (local address 127.0.196.195:39845)
0925 19:05:20.713244 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:20.713257 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:20.713317 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:20.713499 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:20.713500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:20.713502 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:20.713578 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.713580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.714372 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.714373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.715190 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.715191 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.715791 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.715792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.715884 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.715887 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:05:20.715962 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 19:05:20.716014 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":122,"threads_started":1}
I20260925 19:05:20.716535 8603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f3a3501ca7406691cc37097a237ad2" candidate_uuid: "0743aa7c73d64ffc8659e88d576805c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81a2855b68448ad9c7d40da4be26ecd" is_pre_election: true
I20260925 19:05:20.716728 8603 raft_consensus.cc:2454] T 44f3a3501ca7406691cc37097a237ad2 P b81a2855b68448ad9c7d40da4be26ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0743aa7c73d64ffc8659e88d576805c0 in term 0.
I20260925 19:05:20.716759 8454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f3a3501ca7406691cc37097a237ad2" candidate_uuid: "0743aa7c73d64ffc8659e88d576805c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df934d2b8daf44219344ce39308d83e5" is_pre_election: true
I20260925 19:05:20.716962 8454 raft_consensus.cc:2454] T 44f3a3501ca7406691cc37097a237ad2 P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0743aa7c73d64ffc8659e88d576805c0 in term 0.
I20260925 19:05:20.717233 8677 leader_election.cc:302] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0743aa7c73d64ffc8659e88d576805c0, b81a2855b68448ad9c7d40da4be26ecd; no voters:
I20260925 19:05:20.717406 8823 raft_consensus.cc:2781] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:05:20.717468 8823 raft_consensus.cc:482] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:05:20.717490 8823 raft_consensus.cc:3037] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:20.718456 8823 raft_consensus.cc:504] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.718628 8823 leader_election.cc:288] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0 [CANDIDATE]: Term 1 election: Requested vote from peers df934d2b8daf44219344ce39308d83e5 (127.0.196.193:46635), b81a2855b68448ad9c7d40da4be26ecd (127.0.196.194:34129)
I20260925 19:05:20.718784 8603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f3a3501ca7406691cc37097a237ad2" candidate_uuid: "0743aa7c73d64ffc8659e88d576805c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81a2855b68448ad9c7d40da4be26ecd"
I20260925 19:05:20.718789 8454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f3a3501ca7406691cc37097a237ad2" candidate_uuid: "0743aa7c73d64ffc8659e88d576805c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df934d2b8daf44219344ce39308d83e5"
I20260925 19:05:20.718868 8603 raft_consensus.cc:3037] T 44f3a3501ca7406691cc37097a237ad2 P b81a2855b68448ad9c7d40da4be26ecd [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:20.718904 8454 raft_consensus.cc:3037] T 44f3a3501ca7406691cc37097a237ad2 P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:20.719595 8603 raft_consensus.cc:2454] T 44f3a3501ca7406691cc37097a237ad2 P b81a2855b68448ad9c7d40da4be26ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0743aa7c73d64ffc8659e88d576805c0 in term 1.
I20260925 19:05:20.719727 8454 raft_consensus.cc:2454] T 44f3a3501ca7406691cc37097a237ad2 P df934d2b8daf44219344ce39308d83e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0743aa7c73d64ffc8659e88d576805c0 in term 1.
I20260925 19:05:20.719765 8677 leader_election.cc:302] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0743aa7c73d64ffc8659e88d576805c0, b81a2855b68448ad9c7d40da4be26ecd; no voters:
I20260925 19:05:20.719877 8823 raft_consensus.cc:2781] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:05:20.720093 8823 raft_consensus.cc:686] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0 [term 1 LEADER]: Becoming Leader. State: Replica: 0743aa7c73d64ffc8659e88d576805c0, State: Running, Role: LEADER
I20260925 19:05:20.720242 8823 consensus_queue.cc:237] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.721271 8305 catalog_manager.cc:5495] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0743aa7c73d64ffc8659e88d576805c0 (127.0.196.195). New cstate: current_term: 1 leader_uuid: "0743aa7c73d64ffc8659e88d576805c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } health_report { overall_health: HEALTHY } } }
I20260925 19:05:20.738091 8816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.735568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60812 (local address 127.0.196.195:44785)
0925 19:05:20.735615 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:20.735617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:20.735627 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:05:20.735700 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:20.735701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:20.735719 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:20.735809 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:05:20.736022 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.736481 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.737270 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.737470 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.737552 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.737740 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:05:20.737853 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:05:20.737954 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:20.737978 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 19:05:20.738000 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:05:20.740159 8603 raft_consensus.cc:1261] T 44f3a3501ca7406691cc37097a237ad2 P b81a2855b68448ad9c7d40da4be26ecd [term 1 FOLLOWER]: Refusing update from remote peer 0743aa7c73d64ffc8659e88d576805c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:05:20.740197 8454 raft_consensus.cc:1261] T 44f3a3501ca7406691cc37097a237ad2 P df934d2b8daf44219344ce39308d83e5 [term 1 FOLLOWER]: Refusing update from remote peer 0743aa7c73d64ffc8659e88d576805c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:05:20.740538 8823 consensus_queue.cc:1035] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:05:20.740695 8827 consensus_queue.cc:1035] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:05:20.742956 8831 mvcc.cc:205] Tried to move back new op lower bound from 7333327342548299776 to 7333327342471020544. Current Snapshot: MvccSnapshot[applied={T|T < 7333327342548299776}]
I20260925 19:05:20.744023 8832 mvcc.cc:205] Tried to move back new op lower bound from 7333327342548299776 to 7333327342471020544. Current Snapshot: MvccSnapshot[applied={T|T < 7333327342548299776}]
I20260925 19:05:20.745584 8306 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:05:20.745692 8306 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:05:20.746467 8306 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 629f983bc1f14546b301430b0bbe9824: Sending DeleteTablet for 3 replicas of tablet 44f3a3501ca7406691cc37097a237ad2
I20260925 19:05:20.746831 8443 tablet_service.cc:1514] Processing DeleteTablet for tablet 44f3a3501ca7406691cc37097a237ad2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:05:20 UTC) from {username='oryx', principal='oryx/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:38254
I20260925 19:05:20.747006 8306 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:05:20.747169 8306 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:05:20.747133 8583 tablet_service.cc:1514] Processing DeleteTablet for tablet 44f3a3501ca7406691cc37097a237ad2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:05:20 UTC) from {username='oryx', principal='oryx/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:36914
I20260925 19:05:20.747367 8840 tablet_replica.cc:330] T 44f3a3501ca7406691cc37097a237ad2 P df934d2b8daf44219344ce39308d83e5: stopping tablet replica
I20260925 19:05:20.747403 8723 tablet_service.cc:1514] Processing DeleteTablet for tablet 44f3a3501ca7406691cc37097a237ad2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:05:20 UTC) from {username='oryx', principal='oryx/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:60802
I20260925 19:05:20.747496 8840 raft_consensus.cc:2229] T 44f3a3501ca7406691cc37097a237ad2 P df934d2b8daf44219344ce39308d83e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:05:20.747615 8840 raft_consensus.cc:2258] T 44f3a3501ca7406691cc37097a237ad2 P df934d2b8daf44219344ce39308d83e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:05:20.747694 8842 tablet_replica.cc:330] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0: stopping tablet replica
I20260925 19:05:20.747805 8842 raft_consensus.cc:2229] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:05:20.747958 8841 tablet_replica.cc:330] T 44f3a3501ca7406691cc37097a237ad2 P b81a2855b68448ad9c7d40da4be26ecd: stopping tablet replica
I20260925 19:05:20.748075 8841 raft_consensus.cc:2229] T 44f3a3501ca7406691cc37097a237ad2 P b81a2855b68448ad9c7d40da4be26ecd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:05:20.748106 8842 raft_consensus.cc:2258] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:05:20.748175 8841 raft_consensus.cc:2258] T 44f3a3501ca7406691cc37097a237ad2 P b81a2855b68448ad9c7d40da4be26ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:05:20.748636 8841 ts_tablet_manager.cc:1905] T 44f3a3501ca7406691cc37097a237ad2 P b81a2855b68448ad9c7d40da4be26ecd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:05:20.749040 8840 ts_tablet_manager.cc:1905] T 44f3a3501ca7406691cc37097a237ad2 P df934d2b8daf44219344ce39308d83e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:05:20.749120 8443 tablet_service.cc:1467] Processing CreateTablet for tablet 5fcfcb2223314f97b600ebcd5ec9eeef (DEFAULT_TABLE table=test-table [id=036bfc08ba5f451a8f78e490996dc032]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:05:20.749270 8443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fcfcb2223314f97b600ebcd5ec9eeef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:20.750663 8841 ts_tablet_manager.cc:1918] T 44f3a3501ca7406691cc37097a237ad2 P b81a2855b68448ad9c7d40da4be26ecd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:05:20.750733 8819 tablet_bootstrap.cc:492] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5: Bootstrap starting.
I20260925 19:05:20.750744 8841 log.cc:1196] T 44f3a3501ca7406691cc37097a237ad2 P b81a2855b68448ad9c7d40da4be26ecd: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-1/wal/wals/44f3a3501ca7406691cc37097a237ad2
I20260925 19:05:20.750815 8583 tablet_service.cc:1467] Processing CreateTablet for tablet 5fcfcb2223314f97b600ebcd5ec9eeef (DEFAULT_TABLE table=test-table [id=036bfc08ba5f451a8f78e490996dc032]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:05:20.750929 8583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fcfcb2223314f97b600ebcd5ec9eeef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:20.751042 8841 ts_tablet_manager.cc:1939] T 44f3a3501ca7406691cc37097a237ad2 P b81a2855b68448ad9c7d40da4be26ecd: Deleting consensus metadata
I20260925 19:05:20.751143 8819 tablet_bootstrap.cc:654] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:20.751626 8293 catalog_manager.cc:4841] TS b81a2855b68448ad9c7d40da4be26ecd (127.0.196.194:34129): tablet 44f3a3501ca7406691cc37097a237ad2 (table test-table [id=25f6b007db9e49ef8acc0b8f754b83c1]) successfully deleted
I20260925 19:05:20.752242 8817 tablet_bootstrap.cc:492] T 5fcfcb2223314f97b600ebcd5ec9eeef P b81a2855b68448ad9c7d40da4be26ecd: Bootstrap starting.
I20260925 19:05:20.752303 8723 tablet_service.cc:1467] Processing CreateTablet for tablet 5fcfcb2223314f97b600ebcd5ec9eeef (DEFAULT_TABLE table=test-table [id=036bfc08ba5f451a8f78e490996dc032]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:05:20.752429 8723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fcfcb2223314f97b600ebcd5ec9eeef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:20.752625 8817 tablet_bootstrap.cc:654] T 5fcfcb2223314f97b600ebcd5ec9eeef P b81a2855b68448ad9c7d40da4be26ecd: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:20.753190 8817 tablet_bootstrap.cc:492] T 5fcfcb2223314f97b600ebcd5ec9eeef P b81a2855b68448ad9c7d40da4be26ecd: No bootstrap required, opened a new log
I20260925 19:05:20.753258 8817 ts_tablet_manager.cc:1397] T 5fcfcb2223314f97b600ebcd5ec9eeef P b81a2855b68448ad9c7d40da4be26ecd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:20.753409 8817 raft_consensus.cc:348] T 5fcfcb2223314f97b600ebcd5ec9eeef P b81a2855b68448ad9c7d40da4be26ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.753512 8817 raft_consensus.cc:374] T 5fcfcb2223314f97b600ebcd5ec9eeef P b81a2855b68448ad9c7d40da4be26ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:20.753541 8817 raft_consensus.cc:729] T 5fcfcb2223314f97b600ebcd5ec9eeef P b81a2855b68448ad9c7d40da4be26ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81a2855b68448ad9c7d40da4be26ecd, State: Initialized, Role: FOLLOWER
I20260925 19:05:20.753585 8817 consensus_queue.cc:260] T 5fcfcb2223314f97b600ebcd5ec9eeef P b81a2855b68448ad9c7d40da4be26ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.753675 8817 ts_tablet_manager.cc:1428] T 5fcfcb2223314f97b600ebcd5ec9eeef P b81a2855b68448ad9c7d40da4be26ecd: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 19:05:20.753888 8818 tablet_bootstrap.cc:492] T 5fcfcb2223314f97b600ebcd5ec9eeef P 0743aa7c73d64ffc8659e88d576805c0: Bootstrap starting.
I20260925 19:05:20.754284 8840 ts_tablet_manager.cc:1918] T 44f3a3501ca7406691cc37097a237ad2 P df934d2b8daf44219344ce39308d83e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:05:20.754371 8840 log.cc:1196] T 44f3a3501ca7406691cc37097a237ad2 P df934d2b8daf44219344ce39308d83e5: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-0/wal/wals/44f3a3501ca7406691cc37097a237ad2
I20260925 19:05:20.754346 8818 tablet_bootstrap.cc:654] T 5fcfcb2223314f97b600ebcd5ec9eeef P 0743aa7c73d64ffc8659e88d576805c0: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:20.754690 8840 ts_tablet_manager.cc:1939] T 44f3a3501ca7406691cc37097a237ad2 P df934d2b8daf44219344ce39308d83e5: Deleting consensus metadata
I20260925 19:05:20.754724 8842 ts_tablet_manager.cc:1905] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:05:20.755139 8290 catalog_manager.cc:4841] TS df934d2b8daf44219344ce39308d83e5 (127.0.196.193:46635): tablet 44f3a3501ca7406691cc37097a237ad2 (table test-table [id=25f6b007db9e49ef8acc0b8f754b83c1]) successfully deleted
I20260925 19:05:20.755445 8818 tablet_bootstrap.cc:492] T 5fcfcb2223314f97b600ebcd5ec9eeef P 0743aa7c73d64ffc8659e88d576805c0: No bootstrap required, opened a new log
I20260925 19:05:20.755492 8818 ts_tablet_manager.cc:1397] T 5fcfcb2223314f97b600ebcd5ec9eeef P 0743aa7c73d64ffc8659e88d576805c0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:05:20.755587 8819 tablet_bootstrap.cc:492] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5: No bootstrap required, opened a new log
I20260925 19:05:20.755625 8819 ts_tablet_manager.cc:1397] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260925 19:05:20.755646 8818 raft_consensus.cc:348] T 5fcfcb2223314f97b600ebcd5ec9eeef P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.755726 8818 raft_consensus.cc:374] T 5fcfcb2223314f97b600ebcd5ec9eeef P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:20.755749 8818 raft_consensus.cc:729] T 5fcfcb2223314f97b600ebcd5ec9eeef P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0743aa7c73d64ffc8659e88d576805c0, State: Initialized, Role: FOLLOWER
I20260925 19:05:20.755801 8818 consensus_queue.cc:260] T 5fcfcb2223314f97b600ebcd5ec9eeef P 0743aa7c73d64ffc8659e88d576805c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.755903 8818 ts_tablet_manager.cc:1428] T 5fcfcb2223314f97b600ebcd5ec9eeef P 0743aa7c73d64ffc8659e88d576805c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:20.755995 8819 raft_consensus.cc:348] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.756074 8819 raft_consensus.cc:374] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:20.756094 8819 raft_consensus.cc:729] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df934d2b8daf44219344ce39308d83e5, State: Initialized, Role: FOLLOWER
I20260925 19:05:20.756134 8819 consensus_queue.cc:260] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.756237 8819 ts_tablet_manager.cc:1428] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:20.756497 8842 ts_tablet_manager.cc:1918] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:05:20.756579 8842 log.cc:1196] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-2/wal/wals/44f3a3501ca7406691cc37097a237ad2
I20260925 19:05:20.756920 8842 ts_tablet_manager.cc:1939] T 44f3a3501ca7406691cc37097a237ad2 P 0743aa7c73d64ffc8659e88d576805c0: Deleting consensus metadata
I20260925 19:05:20.757355 8293 catalog_manager.cc:4841] TS 0743aa7c73d64ffc8659e88d576805c0 (127.0.196.195:44785): tablet 44f3a3501ca7406691cc37097a237ad2 (table test-table [id=25f6b007db9e49ef8acc0b8f754b83c1]) successfully deleted
I20260925 19:05:20.781725 8825 raft_consensus.cc:482] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:05:20.781822 8825 raft_consensus.cc:504] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.782226 8825 leader_election.cc:288] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b81a2855b68448ad9c7d40da4be26ecd (127.0.196.194:34129), 0743aa7c73d64ffc8659e88d576805c0 (127.0.196.195:44785)
I20260925 19:05:20.786360 8816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.782727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:44495 (local address 127.0.196.195:44785)
0925 19:05:20.782768 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:20.782770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:20.782946 (+ 176us) server_negotiation.cc:407] Connection header received
0925 19:05:20.782992 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:20.782993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:20.783004 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:20.783083 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:20.783275 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.784104 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.785415 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.785945 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.786046 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.786058 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:20.786257 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 19:05:20.786291 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:05:20.786365 8814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.782506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:45265 (local address 127.0.196.194:34129)
0925 19:05:20.782563 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:20.782566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:20.782789 (+ 223us) server_negotiation.cc:407] Connection header received
0925 19:05:20.783142 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:20.783143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:20.783153 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:20.783205 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:20.783388 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.783868 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.785370 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.786009 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.786092 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.786101 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:20.786236 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 19:05:20.786282 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 19:05:20.786396 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.782667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:44785 (local address 127.0.196.193:44495)
0925 19:05:20.782832 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:20.782841 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:20.782940 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:20.783115 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:20.783115 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:20.783118 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:20.783204 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.783206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.784188 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.784189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.785260 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.785263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.786064 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.786065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.786150 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.786156 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 19:05:20.786216 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 19:05:20.786284 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
I20260925 19:05:20.786743 8846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.782418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:34129 (local address 127.0.196.193:45265)
0925 19:05:20.782688 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:20.782701 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:20.782810 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:20.783224 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:20.783225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:20.783226 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:20.783299 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.783300 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.783969 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.783970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.785260 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.785263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.786105 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.786105 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.786163 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.786166 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:05:20.786216 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 19:05:20.786667 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":386,"thread_start_us":85,"threads_started":1}
I20260925 19:05:20.786670 8743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcfcb2223314f97b600ebcd5ec9eeef" candidate_uuid: "df934d2b8daf44219344ce39308d83e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0743aa7c73d64ffc8659e88d576805c0" is_pre_election: true
I20260925 19:05:20.786804 8743 raft_consensus.cc:2454] T 5fcfcb2223314f97b600ebcd5ec9eeef P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df934d2b8daf44219344ce39308d83e5 in term 0.
I20260925 19:05:20.786960 8603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcfcb2223314f97b600ebcd5ec9eeef" candidate_uuid: "df934d2b8daf44219344ce39308d83e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81a2855b68448ad9c7d40da4be26ecd" is_pre_election: true
I20260925 19:05:20.787045 8603 raft_consensus.cc:2454] T 5fcfcb2223314f97b600ebcd5ec9eeef P b81a2855b68448ad9c7d40da4be26ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df934d2b8daf44219344ce39308d83e5 in term 0.
I20260925 19:05:20.787009 8397 leader_election.cc:302] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0743aa7c73d64ffc8659e88d576805c0, df934d2b8daf44219344ce39308d83e5; no voters:
I20260925 19:05:20.787129 8825 raft_consensus.cc:2781] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:05:20.787164 8825 raft_consensus.cc:482] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:05:20.787190 8825 raft_consensus.cc:3037] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:20.787876 8825 raft_consensus.cc:504] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.788023 8825 leader_election.cc:288] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5 [CANDIDATE]: Term 1 election: Requested vote from peers b81a2855b68448ad9c7d40da4be26ecd (127.0.196.194:34129), 0743aa7c73d64ffc8659e88d576805c0 (127.0.196.195:44785)
I20260925 19:05:20.788187 8603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcfcb2223314f97b600ebcd5ec9eeef" candidate_uuid: "df934d2b8daf44219344ce39308d83e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81a2855b68448ad9c7d40da4be26ecd"
I20260925 19:05:20.788226 8743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcfcb2223314f97b600ebcd5ec9eeef" candidate_uuid: "df934d2b8daf44219344ce39308d83e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0743aa7c73d64ffc8659e88d576805c0"
I20260925 19:05:20.788285 8603 raft_consensus.cc:3037] T 5fcfcb2223314f97b600ebcd5ec9eeef P b81a2855b68448ad9c7d40da4be26ecd [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:20.788316 8743 raft_consensus.cc:3037] T 5fcfcb2223314f97b600ebcd5ec9eeef P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:20.788921 8603 raft_consensus.cc:2454] T 5fcfcb2223314f97b600ebcd5ec9eeef P b81a2855b68448ad9c7d40da4be26ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df934d2b8daf44219344ce39308d83e5 in term 1.
I20260925 19:05:20.789096 8397 leader_election.cc:302] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b81a2855b68448ad9c7d40da4be26ecd, df934d2b8daf44219344ce39308d83e5; no voters:
I20260925 19:05:20.789230 8825 raft_consensus.cc:2781] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:05:20.789232 8743 raft_consensus.cc:2454] T 5fcfcb2223314f97b600ebcd5ec9eeef P 0743aa7c73d64ffc8659e88d576805c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df934d2b8daf44219344ce39308d83e5 in term 1.
I20260925 19:05:20.789392 8825 raft_consensus.cc:686] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5 [term 1 LEADER]: Becoming Leader. State: Replica: df934d2b8daf44219344ce39308d83e5, State: Running, Role: LEADER
I20260925 19:05:20.789497 8825 consensus_queue.cc:237] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.790191 8306 catalog_manager.cc:5495] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to df934d2b8daf44219344ce39308d83e5 (127.0.196.193). New cstate: current_term: 1 leader_uuid: "df934d2b8daf44219344ce39308d83e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } health_report { overall_health: UNKNOWN } } }
W20260925 19:05:20.813694 8790 tablet.cc:2369] T 5fcfcb2223314f97b600ebcd5ec9eeef P 0743aa7c73d64ffc8659e88d576805c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 19:05:20 dist-test-slave-rn98 krb5kdc[8261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363119, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.254@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM
I20260925 19:05:20.836113 8855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.827608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41331 (local address 127.0.0.1:43734)
0925 19:05:20.827786 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:20.827797 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:20.827899 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:20.828080 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:20.828081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:20.828509 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:20.828605 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.828608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.829331 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.829332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.829921 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.829922 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.830315 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.830316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.830372 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.831125 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:20.831132 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:20.832901 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:20.834794 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:20.834802 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:20.834805 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:20.835042 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:20.835270 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:20.835271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:20.835271 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:20.835336 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:20.835605 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:20.835612 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:20.835661 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 19:05:20.835751 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 19:05:20.835949 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":97,"threads_started":1}
I20260925 19:05:20.836184 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.827667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43734 (local address 127.0.196.254:41331)
0925 19:05:20.827742 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:20.827745 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:20.827877 (+ 132us) server_negotiation.cc:407] Connection header received
0925 19:05:20.827981 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:20.827982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:20.827991 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:20.828076 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:20.828690 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.829242 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.830022 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.830223 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.830314 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.833029 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:20.833031 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:20.833031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:20.833039 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:20.834609 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:20.835139 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:20.835140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:20.835140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:20.835199 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:20.835406 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:20.835407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:20.835408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:20.835533 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:20.835617 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:20.835807 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 19:05:20.836009 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":125,"server-negotiator.queue_time_us":44}
I20260925 19:05:20.838660 8306 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:43734:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 19:05:20.838896 8306 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:05:20.840585 8443 tablet_service.cc:1467] Processing CreateTablet for tablet b48f62a6491645b7a3a71a60653c6020 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=294aec83a08b479a9225c56a2b1a1df9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 19:05:20.840641 8583 tablet_service.cc:1467] Processing CreateTablet for tablet b48f62a6491645b7a3a71a60653c6020 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=294aec83a08b479a9225c56a2b1a1df9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 19:05:20.840723 8443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b48f62a6491645b7a3a71a60653c6020. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:20.840804 8583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b48f62a6491645b7a3a71a60653c6020. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:20.842088 8819 tablet_bootstrap.cc:492] T b48f62a6491645b7a3a71a60653c6020 P df934d2b8daf44219344ce39308d83e5: Bootstrap starting.
I20260925 19:05:20.842510 8817 tablet_bootstrap.cc:492] T b48f62a6491645b7a3a71a60653c6020 P b81a2855b68448ad9c7d40da4be26ecd: Bootstrap starting.
I20260925 19:05:20.842643 8819 tablet_bootstrap.cc:654] T b48f62a6491645b7a3a71a60653c6020 P df934d2b8daf44219344ce39308d83e5: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:20.842842 8723 tablet_service.cc:1467] Processing CreateTablet for tablet b48f62a6491645b7a3a71a60653c6020 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=294aec83a08b479a9225c56a2b1a1df9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 19:05:20.843007 8723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b48f62a6491645b7a3a71a60653c6020. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:20.843000 8817 tablet_bootstrap.cc:654] T b48f62a6491645b7a3a71a60653c6020 P b81a2855b68448ad9c7d40da4be26ecd: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:20.843698 8819 tablet_bootstrap.cc:492] T b48f62a6491645b7a3a71a60653c6020 P df934d2b8daf44219344ce39308d83e5: No bootstrap required, opened a new log
I20260925 19:05:20.843700 8817 tablet_bootstrap.cc:492] T b48f62a6491645b7a3a71a60653c6020 P b81a2855b68448ad9c7d40da4be26ecd: No bootstrap required, opened a new log
I20260925 19:05:20.843748 8817 ts_tablet_manager.cc:1397] T b48f62a6491645b7a3a71a60653c6020 P b81a2855b68448ad9c7d40da4be26ecd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:20.843748 8819 ts_tablet_manager.cc:1397] T b48f62a6491645b7a3a71a60653c6020 P df934d2b8daf44219344ce39308d83e5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:05:20.843923 8817 raft_consensus.cc:348] T b48f62a6491645b7a3a71a60653c6020 P b81a2855b68448ad9c7d40da4be26ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.843918 8819 raft_consensus.cc:348] T b48f62a6491645b7a3a71a60653c6020 P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.843995 8817 raft_consensus.cc:374] T b48f62a6491645b7a3a71a60653c6020 P b81a2855b68448ad9c7d40da4be26ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:20.843999 8819 raft_consensus.cc:374] T b48f62a6491645b7a3a71a60653c6020 P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:20.844023 8817 raft_consensus.cc:729] T b48f62a6491645b7a3a71a60653c6020 P b81a2855b68448ad9c7d40da4be26ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81a2855b68448ad9c7d40da4be26ecd, State: Initialized, Role: FOLLOWER
I20260925 19:05:20.844024 8819 raft_consensus.cc:729] T b48f62a6491645b7a3a71a60653c6020 P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df934d2b8daf44219344ce39308d83e5, State: Initialized, Role: FOLLOWER
I20260925 19:05:20.844072 8819 consensus_queue.cc:260] T b48f62a6491645b7a3a71a60653c6020 P df934d2b8daf44219344ce39308d83e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.844072 8817 consensus_queue.cc:260] T b48f62a6491645b7a3a71a60653c6020 P b81a2855b68448ad9c7d40da4be26ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.844162 8817 ts_tablet_manager.cc:1428] T b48f62a6491645b7a3a71a60653c6020 P b81a2855b68448ad9c7d40da4be26ecd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:20.844177 8819 ts_tablet_manager.cc:1428] T b48f62a6491645b7a3a71a60653c6020 P df934d2b8daf44219344ce39308d83e5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 19:05:20.844338 8822 tablet_replica.cc:439] T b48f62a6491645b7a3a71a60653c6020 P b81a2855b68448ad9c7d40da4be26ecd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } } }
I20260925 19:05:20.844405 8822 tablet_replica.cc:442] T b48f62a6491645b7a3a71a60653c6020 P b81a2855b68448ad9c7d40da4be26ecd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:05:20.844493 8825 tablet_replica.cc:439] T b48f62a6491645b7a3a71a60653c6020 P df934d2b8daf44219344ce39308d83e5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } } }
I20260925 19:05:20.844575 8818 tablet_bootstrap.cc:492] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0: Bootstrap starting.
I20260925 19:05:20.844591 8825 tablet_replica.cc:442] T b48f62a6491645b7a3a71a60653c6020 P df934d2b8daf44219344ce39308d83e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:05:20.845017 8818 tablet_bootstrap.cc:654] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:20.845955 8818 tablet_bootstrap.cc:492] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0: No bootstrap required, opened a new log
I20260925 19:05:20.846000 8818 ts_tablet_manager.cc:1397] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:20.846163 8818 raft_consensus.cc:348] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.846237 8818 raft_consensus.cc:374] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:20.846252 8818 raft_consensus.cc:729] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0743aa7c73d64ffc8659e88d576805c0, State: Initialized, Role: FOLLOWER
I20260925 19:05:20.846309 8818 consensus_queue.cc:260] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.846421 8818 ts_tablet_manager.cc:1428] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:20.846530 8837 tablet_replica.cc:439] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } } }
I20260925 19:05:20.846642 8837 tablet_replica.cc:442] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:05:20.876626 8837 raft_consensus.cc:482] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:05:20.876726 8837 raft_consensus.cc:504] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.876885 8837 leader_election.cc:288] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df934d2b8daf44219344ce39308d83e5 (127.0.196.193:46635), b81a2855b68448ad9c7d40da4be26ecd (127.0.196.194:34129)
I20260925 19:05:20.877120 8454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48f62a6491645b7a3a71a60653c6020" candidate_uuid: "0743aa7c73d64ffc8659e88d576805c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df934d2b8daf44219344ce39308d83e5" is_pre_election: true
I20260925 19:05:20.877149 8603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48f62a6491645b7a3a71a60653c6020" candidate_uuid: "0743aa7c73d64ffc8659e88d576805c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81a2855b68448ad9c7d40da4be26ecd" is_pre_election: true
I20260925 19:05:20.877240 8454 raft_consensus.cc:2454] T b48f62a6491645b7a3a71a60653c6020 P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0743aa7c73d64ffc8659e88d576805c0 in term 0.
I20260925 19:05:20.877262 8603 raft_consensus.cc:2454] T b48f62a6491645b7a3a71a60653c6020 P b81a2855b68448ad9c7d40da4be26ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0743aa7c73d64ffc8659e88d576805c0 in term 0.
I20260925 19:05:20.877431 8674 leader_election.cc:302] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0743aa7c73d64ffc8659e88d576805c0, df934d2b8daf44219344ce39308d83e5; no voters:
I20260925 19:05:20.877549 8837 raft_consensus.cc:2781] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:05:20.877589 8837 raft_consensus.cc:482] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:05:20.877619 8837 raft_consensus.cc:3037] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:20.878312 8837 raft_consensus.cc:504] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.878464 8837 leader_election.cc:288] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0 [CANDIDATE]: Term 1 election: Requested vote from peers df934d2b8daf44219344ce39308d83e5 (127.0.196.193:46635), b81a2855b68448ad9c7d40da4be26ecd (127.0.196.194:34129)
I20260925 19:05:20.878640 8603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48f62a6491645b7a3a71a60653c6020" candidate_uuid: "0743aa7c73d64ffc8659e88d576805c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81a2855b68448ad9c7d40da4be26ecd"
I20260925 19:05:20.878665 8454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48f62a6491645b7a3a71a60653c6020" candidate_uuid: "0743aa7c73d64ffc8659e88d576805c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df934d2b8daf44219344ce39308d83e5"
I20260925 19:05:20.878716 8603 raft_consensus.cc:3037] T b48f62a6491645b7a3a71a60653c6020 P b81a2855b68448ad9c7d40da4be26ecd [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:20.878724 8454 raft_consensus.cc:3037] T b48f62a6491645b7a3a71a60653c6020 P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:20.879558 8603 raft_consensus.cc:2454] T b48f62a6491645b7a3a71a60653c6020 P b81a2855b68448ad9c7d40da4be26ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0743aa7c73d64ffc8659e88d576805c0 in term 1.
I20260925 19:05:20.879558 8454 raft_consensus.cc:2454] T b48f62a6491645b7a3a71a60653c6020 P df934d2b8daf44219344ce39308d83e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0743aa7c73d64ffc8659e88d576805c0 in term 1.
I20260925 19:05:20.879762 8674 leader_election.cc:302] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0743aa7c73d64ffc8659e88d576805c0, df934d2b8daf44219344ce39308d83e5; no voters:
I20260925 19:05:20.879853 8837 raft_consensus.cc:2781] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:05:20.879940 8837 raft_consensus.cc:686] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0 [term 1 LEADER]: Becoming Leader. State: Replica: 0743aa7c73d64ffc8659e88d576805c0, State: Running, Role: LEADER
I20260925 19:05:20.879994 8837 consensus_queue.cc:237] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.880350 8827 tablet_replica.cc:439] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0: TxnStatusTablet state changed. Reason: New leader 0743aa7c73d64ffc8659e88d576805c0. Latest consensus state: current_term: 1 leader_uuid: "0743aa7c73d64ffc8659e88d576805c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } } }
I20260925 19:05:20.880443 8827 tablet_replica.cc:442] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0: This TxnStatusTablet replica's current role is: LEADER
I20260925 19:05:20.880726 8306 catalog_manager.cc:5495] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0743aa7c73d64ffc8659e88d576805c0 (127.0.196.195). New cstate: current_term: 1 leader_uuid: "0743aa7c73d64ffc8659e88d576805c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } health_report { overall_health: HEALTHY } } }
I20260925 19:05:20.881040 8860 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 19:05:20.881158 8860 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 19:05:20.886264 8816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.883833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60814 (local address 127.0.196.195:44785)
0925 19:05:20.883879 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:20.883881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:20.883891 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:05:20.883990 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:20.883991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:20.884003 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:20.884056 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:05:20.884167 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.884684 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.885515 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.885732 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.885817 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.885999 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:05:20.886036 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:05:20.886090 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:20.886167 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 19:05:20.886186 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:05:20.886291 8855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.883758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:44785 (local address 127.0.0.1:60814)
0925 19:05:20.883833 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:20.883846 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:20.883910 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:20.884059 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:20.884059 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:20.884061 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:05:20.884113 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.884115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.884772 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.884773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.885402 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.885404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.885818 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.885819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.885894 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.885910 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:05:20.886097 (+ 187us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:05:20.886103 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 19:05:20.886161 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 19:05:20.886212 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 19:05:20.887323 8603 raft_consensus.cc:1261] T b48f62a6491645b7a3a71a60653c6020 P b81a2855b68448ad9c7d40da4be26ecd [term 1 FOLLOWER]: Refusing update from remote peer 0743aa7c73d64ffc8659e88d576805c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:05:20.887342 8454 raft_consensus.cc:1261] T b48f62a6491645b7a3a71a60653c6020 P df934d2b8daf44219344ce39308d83e5 [term 1 FOLLOWER]: Refusing update from remote peer 0743aa7c73d64ffc8659e88d576805c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:05:20.887552 8827 consensus_queue.cc:1035] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:05:20.887676 8837 consensus_queue.cc:1035] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:05:20.888041 8822 tablet_replica.cc:439] T b48f62a6491645b7a3a71a60653c6020 P b81a2855b68448ad9c7d40da4be26ecd: TxnStatusTablet state changed. Reason: New leader 0743aa7c73d64ffc8659e88d576805c0. Latest consensus state: current_term: 1 leader_uuid: "0743aa7c73d64ffc8659e88d576805c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } } }
I20260925 19:05:20.888155 8822 tablet_replica.cc:442] T b48f62a6491645b7a3a71a60653c6020 P b81a2855b68448ad9c7d40da4be26ecd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:05:20.888470 8825 tablet_replica.cc:439] T b48f62a6491645b7a3a71a60653c6020 P df934d2b8daf44219344ce39308d83e5: TxnStatusTablet state changed. Reason: New leader 0743aa7c73d64ffc8659e88d576805c0. Latest consensus state: current_term: 1 leader_uuid: "0743aa7c73d64ffc8659e88d576805c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } } }
I20260925 19:05:20.888566 8825 tablet_replica.cc:442] T b48f62a6491645b7a3a71a60653c6020 P df934d2b8daf44219344ce39308d83e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:05:20.888832 8837 tablet_replica.cc:439] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0743aa7c73d64ffc8659e88d576805c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } } }
I20260925 19:05:20.888934 8837 tablet_replica.cc:442] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0: This TxnStatusTablet replica's current role is: LEADER
I20260925 19:05:20.889073 8830 mvcc.cc:205] Tried to move back new op lower bound from 7333327343152844800 to 7333327343124877312. Current Snapshot: MvccSnapshot[applied={T|T < 7333327343152844800 or (T in {7333327343152844800})}]
I20260925 19:05:20.889580 8823 tablet_replica.cc:439] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0743aa7c73d64ffc8659e88d576805c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } } }
I20260925 19:05:20.889664 8823 tablet_replica.cc:442] T b48f62a6491645b7a3a71a60653c6020 P 0743aa7c73d64ffc8659e88d576805c0: This TxnStatusTablet replica's current role is: LEADER
I20260925 19:05:20.890162 8822 tablet_replica.cc:439] T b48f62a6491645b7a3a71a60653c6020 P b81a2855b68448ad9c7d40da4be26ecd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0743aa7c73d64ffc8659e88d576805c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } } }
I20260925 19:05:20.890259 8822 tablet_replica.cc:442] T b48f62a6491645b7a3a71a60653c6020 P b81a2855b68448ad9c7d40da4be26ecd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:05:20.890967 8825 tablet_replica.cc:439] T b48f62a6491645b7a3a71a60653c6020 P df934d2b8daf44219344ce39308d83e5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0743aa7c73d64ffc8659e88d576805c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } } }
I20260925 19:05:20.891067 8825 tablet_replica.cc:442] T b48f62a6491645b7a3a71a60653c6020 P df934d2b8daf44219344ce39308d83e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:05:20.893401 8813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.890466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38256 (local address 127.0.196.193:46635)
0925 19:05:20.890507 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:20.890509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:20.890517 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:05:20.890555 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:20.890556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:20.890577 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:20.890698 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:05:20.890837 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.891599 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.892511 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.892741 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.892822 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.893020 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:05:20.893146 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:05:20.893238 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:20.893281 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 19:05:20.893303 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:05:20.899627 8816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.896496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:59397 (local address 127.0.196.195:44785)
0925 19:05:20.896576 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:20.896578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:20.896724 (+ 146us) server_negotiation.cc:407] Connection header received
0925 19:05:20.896784 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:20.896785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:20.896795 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:20.896854 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:05:20.897022 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.897574 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.898925 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.899133 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.899193 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.899404 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:05:20.899430 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:05:20.899465 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:20.899552 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 19:05:20.899568 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:05:20.899717 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.896334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:44785 (local address 127.0.196.193:59397)
0925 19:05:20.896592 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:20.896642 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:20.896726 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:20.896858 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:20.896858 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:20.896860 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:05:20.896946 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.896948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.897670 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.897671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.898830 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.898834 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.899216 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.899217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.899293 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.899313 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:05:20.899489 (+ 176us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:05:20.899491 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:05:20.899551 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 19:05:20.899599 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":78,"threads_started":1}
I20260925 19:05:20.901978 8603 raft_consensus.cc:1261] T 5fcfcb2223314f97b600ebcd5ec9eeef P b81a2855b68448ad9c7d40da4be26ecd [term 1 FOLLOWER]: Refusing update from remote peer df934d2b8daf44219344ce39308d83e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:05:20.902235 8743 raft_consensus.cc:1261] T 5fcfcb2223314f97b600ebcd5ec9eeef P 0743aa7c73d64ffc8659e88d576805c0 [term 1 FOLLOWER]: Refusing update from remote peer df934d2b8daf44219344ce39308d83e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:05:20.902310 8825 consensus_queue.cc:1035] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:05:20.902505 8848 consensus_queue.cc:1035] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:05:20.914078 8795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.911007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:46635 (local address 127.0.196.195:59755)
0925 19:05:20.911071 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:20.911085 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:20.911131 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:20.911361 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:20.911362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:20.911364 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:05:20.911417 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.911419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.912418 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.912420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.913076 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:20.913078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.913572 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.913573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:20.913647 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.913668 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:05:20.913921 (+ 253us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:05:20.913924 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:05:20.913972 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 19:05:20.914016 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 19:05:20.914093 8813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.911118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:59755 (local address 127.0.196.193:46635)
0925 19:05:20.911211 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:20.911214 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:20.911223 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:05:20.911245 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:20.911246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:20.911258 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:20.911310 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:05:20.911486 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.912325 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.913162 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.913498 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.913775 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.913802 (+ 27us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:05:20.913864 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:05:20.913915 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:20.913983 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 19:05:20.914008 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 19:05:20.921801 8306 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:05:20.921933 8306 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:05:20.922585 8306 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 629f983bc1f14546b301430b0bbe9824: Sending DeleteTablet for 3 replicas of tablet 5fcfcb2223314f97b600ebcd5ec9eeef
I20260925 19:05:20.922984 8723 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fcfcb2223314f97b600ebcd5ec9eeef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:05:20 UTC) from {username='oryx', principal='oryx/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:60802
I20260925 19:05:20.923032 8443 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fcfcb2223314f97b600ebcd5ec9eeef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:05:20 UTC) from {username='oryx', principal='oryx/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:38254
I20260925 19:05:20.923105 8842 tablet_replica.cc:330] T 5fcfcb2223314f97b600ebcd5ec9eeef P 0743aa7c73d64ffc8659e88d576805c0: stopping tablet replica
I20260925 19:05:20.923127 8583 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fcfcb2223314f97b600ebcd5ec9eeef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:05:20 UTC) from {username='oryx', principal='oryx/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:36914
I20260925 19:05:20.923185 8842 raft_consensus.cc:2229] T 5fcfcb2223314f97b600ebcd5ec9eeef P 0743aa7c73d64ffc8659e88d576805c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:05:20.923214 8840 tablet_replica.cc:330] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5: stopping tablet replica
I20260925 19:05:20.923226 8842 raft_consensus.cc:2258] T 5fcfcb2223314f97b600ebcd5ec9eeef P 0743aa7c73d64ffc8659e88d576805c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:05:20.923267 8840 raft_consensus.cc:2229] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:05:20.923373 8840 raft_consensus.cc:2258] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:05:20.923461 8841 tablet_replica.cc:330] T 5fcfcb2223314f97b600ebcd5ec9eeef P b81a2855b68448ad9c7d40da4be26ecd: stopping tablet replica
I20260925 19:05:20.923524 8841 raft_consensus.cc:2229] T 5fcfcb2223314f97b600ebcd5ec9eeef P b81a2855b68448ad9c7d40da4be26ecd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:05:20.923574 8841 raft_consensus.cc:2258] T 5fcfcb2223314f97b600ebcd5ec9eeef P b81a2855b68448ad9c7d40da4be26ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:05:20.924011 8840 ts_tablet_manager.cc:1905] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:05:20.924154 8842 ts_tablet_manager.cc:1905] T 5fcfcb2223314f97b600ebcd5ec9eeef P 0743aa7c73d64ffc8659e88d576805c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:05:20.925915 8840 ts_tablet_manager.cc:1918] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:05:20.925982 8840 log.cc:1196] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-0/wal/wals/5fcfcb2223314f97b600ebcd5ec9eeef
I20260925 19:05:20.926221 8840 ts_tablet_manager.cc:1939] T 5fcfcb2223314f97b600ebcd5ec9eeef P df934d2b8daf44219344ce39308d83e5: Deleting consensus metadata
I20260925 19:05:20.928454 8290 catalog_manager.cc:4841] TS df934d2b8daf44219344ce39308d83e5 (127.0.196.193:46635): tablet 5fcfcb2223314f97b600ebcd5ec9eeef (table test-table [id=036bfc08ba5f451a8f78e490996dc032]) successfully deleted
I20260925 19:05:20.929386 8842 ts_tablet_manager.cc:1918] T 5fcfcb2223314f97b600ebcd5ec9eeef P 0743aa7c73d64ffc8659e88d576805c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:05:20.929476 8842 log.cc:1196] T 5fcfcb2223314f97b600ebcd5ec9eeef P 0743aa7c73d64ffc8659e88d576805c0: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-2/wal/wals/5fcfcb2223314f97b600ebcd5ec9eeef
I20260925 19:05:20.929699 8842 ts_tablet_manager.cc:1939] T 5fcfcb2223314f97b600ebcd5ec9eeef P 0743aa7c73d64ffc8659e88d576805c0: Deleting consensus metadata
I20260925 19:05:20.930079 8293 catalog_manager.cc:4841] TS 0743aa7c73d64ffc8659e88d576805c0 (127.0.196.195:44785): tablet 5fcfcb2223314f97b600ebcd5ec9eeef (table test-table [id=036bfc08ba5f451a8f78e490996dc032]) successfully deleted
I20260925 19:05:20.931629 8841 ts_tablet_manager.cc:1905] T 5fcfcb2223314f97b600ebcd5ec9eeef P b81a2855b68448ad9c7d40da4be26ecd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:05:20.933677 8841 ts_tablet_manager.cc:1918] T 5fcfcb2223314f97b600ebcd5ec9eeef P b81a2855b68448ad9c7d40da4be26ecd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:05:20.933736 8841 log.cc:1196] T 5fcfcb2223314f97b600ebcd5ec9eeef P b81a2855b68448ad9c7d40da4be26ecd: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-1/wal/wals/5fcfcb2223314f97b600ebcd5ec9eeef
I20260925 19:05:20.933969 8841 ts_tablet_manager.cc:1939] T 5fcfcb2223314f97b600ebcd5ec9eeef P b81a2855b68448ad9c7d40da4be26ecd: Deleting consensus metadata
I20260925 19:05:20.934523 8293 catalog_manager.cc:4841] TS b81a2855b68448ad9c7d40da4be26ecd (127.0.196.194:34129): tablet 5fcfcb2223314f97b600ebcd5ec9eeef (table test-table [id=036bfc08ba5f451a8f78e490996dc032]) successfully deleted
Sep 25 19:05:20 dist-test-slave-rn98 krb5kdc[8261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 19:05:20 dist-test-slave-rn98 krb5kdc[8261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:20 dist-test-slave-rn98 krb5kdc[8261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 19:05:20.952625 8356 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:20.946558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43750 (local address 127.0.196.254:41331)
0925 19:05:20.946611 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:20.946613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:20.946797 (+ 184us) server_negotiation.cc:407] Connection header received
0925 19:05:20.946882 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:20.946883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:20.946896 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:20.946985 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:20.947624 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.948167 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.949024 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.949244 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.949326 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.952571 (+ 3245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43750: BlockingRecv error: recv got EOF from 127.0.0.1:43750 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 25 19:05:20 dist-test-slave-rn98 krb5kdc[8261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM
I20260925 19:05:20.962532 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.954080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43752 (local address 127.0.196.254:41331)
0925 19:05:20.954127 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:20.954129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:20.954248 (+ 119us) server_negotiation.cc:407] Connection header received
0925 19:05:20.954346 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:20.954347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:20.954358 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:20.954422 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:20.955065 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.955698 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.956590 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.956859 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.956946 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.959572 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:20.959574 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:20.959575 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:20.959584 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:20.961224 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:20.961659 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:20.961661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:20.961661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:20.961728 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:20.961946 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:20.961948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:20.961948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:20.962082 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:20.962202 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:20.962312 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 19:05:20.962407 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:05:20.964450 8306 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:05:20.964636 8306 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:05:20.966109 8583 tablet_service.cc:1467] Processing CreateTablet for tablet 6a6134c7393c49dda0906f7b0bc59ab6 (DEFAULT_TABLE table=test-table [id=0a641d375a534f0b8c60f30cc055a572]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:05:20.966109 8723 tablet_service.cc:1467] Processing CreateTablet for tablet 6a6134c7393c49dda0906f7b0bc59ab6 (DEFAULT_TABLE table=test-table [id=0a641d375a534f0b8c60f30cc055a572]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:05:20.966251 8723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a6134c7393c49dda0906f7b0bc59ab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:20.966251 8583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a6134c7393c49dda0906f7b0bc59ab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:20.967777 8443 tablet_service.cc:1467] Processing CreateTablet for tablet 6a6134c7393c49dda0906f7b0bc59ab6 (DEFAULT_TABLE table=test-table [id=0a641d375a534f0b8c60f30cc055a572]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:05:20.967895 8443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a6134c7393c49dda0906f7b0bc59ab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:20.968839 8819 tablet_bootstrap.cc:492] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5: Bootstrap starting.
I20260925 19:05:20.968901 8817 tablet_bootstrap.cc:492] T 6a6134c7393c49dda0906f7b0bc59ab6 P b81a2855b68448ad9c7d40da4be26ecd: Bootstrap starting.
I20260925 19:05:20.969314 8819 tablet_bootstrap.cc:654] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:20.969313 8817 tablet_bootstrap.cc:654] T 6a6134c7393c49dda0906f7b0bc59ab6 P b81a2855b68448ad9c7d40da4be26ecd: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:20.969720 8818 tablet_bootstrap.cc:492] T 6a6134c7393c49dda0906f7b0bc59ab6 P 0743aa7c73d64ffc8659e88d576805c0: Bootstrap starting.
I20260925 19:05:20.970022 8819 tablet_bootstrap.cc:492] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5: No bootstrap required, opened a new log
I20260925 19:05:20.970072 8819 ts_tablet_manager.cc:1397] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:20.970078 8818 tablet_bootstrap.cc:654] T 6a6134c7393c49dda0906f7b0bc59ab6 P 0743aa7c73d64ffc8659e88d576805c0: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:20.970258 8819 raft_consensus.cc:348] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.970345 8819 raft_consensus.cc:374] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:20.970374 8819 raft_consensus.cc:729] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df934d2b8daf44219344ce39308d83e5, State: Initialized, Role: FOLLOWER
I20260925 19:05:20.970439 8819 consensus_queue.cc:260] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.970562 8819 ts_tablet_manager.cc:1428] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 19:05:20.972380 8817 tablet_bootstrap.cc:492] T 6a6134c7393c49dda0906f7b0bc59ab6 P b81a2855b68448ad9c7d40da4be26ecd: No bootstrap required, opened a new log
I20260925 19:05:20.972380 8818 tablet_bootstrap.cc:492] T 6a6134c7393c49dda0906f7b0bc59ab6 P 0743aa7c73d64ffc8659e88d576805c0: No bootstrap required, opened a new log
I20260925 19:05:20.972440 8817 ts_tablet_manager.cc:1397] T 6a6134c7393c49dda0906f7b0bc59ab6 P b81a2855b68448ad9c7d40da4be26ecd: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 19:05:20.972440 8818 ts_tablet_manager.cc:1397] T 6a6134c7393c49dda0906f7b0bc59ab6 P 0743aa7c73d64ffc8659e88d576805c0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 19:05:20.972566 8818 raft_consensus.cc:348] T 6a6134c7393c49dda0906f7b0bc59ab6 P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.972638 8818 raft_consensus.cc:374] T 6a6134c7393c49dda0906f7b0bc59ab6 P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:20.972602 8817 raft_consensus.cc:348] T 6a6134c7393c49dda0906f7b0bc59ab6 P b81a2855b68448ad9c7d40da4be26ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.972661 8818 raft_consensus.cc:729] T 6a6134c7393c49dda0906f7b0bc59ab6 P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0743aa7c73d64ffc8659e88d576805c0, State: Initialized, Role: FOLLOWER
I20260925 19:05:20.972671 8817 raft_consensus.cc:374] T 6a6134c7393c49dda0906f7b0bc59ab6 P b81a2855b68448ad9c7d40da4be26ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:20.972695 8817 raft_consensus.cc:729] T 6a6134c7393c49dda0906f7b0bc59ab6 P b81a2855b68448ad9c7d40da4be26ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81a2855b68448ad9c7d40da4be26ecd, State: Initialized, Role: FOLLOWER
I20260925 19:05:20.972711 8818 consensus_queue.cc:260] T 6a6134c7393c49dda0906f7b0bc59ab6 P 0743aa7c73d64ffc8659e88d576805c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.972738 8817 consensus_queue.cc:260] T 6a6134c7393c49dda0906f7b0bc59ab6 P b81a2855b68448ad9c7d40da4be26ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.972813 8818 ts_tablet_manager.cc:1428] T 6a6134c7393c49dda0906f7b0bc59ab6 P 0743aa7c73d64ffc8659e88d576805c0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 19:05:20.972829 8817 ts_tablet_manager.cc:1428] T 6a6134c7393c49dda0906f7b0bc59ab6 P b81a2855b68448ad9c7d40da4be26ecd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:20.976954 8871 raft_consensus.cc:482] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:05:20.977056 8871 raft_consensus.cc:504] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.977247 8871 leader_election.cc:288] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b81a2855b68448ad9c7d40da4be26ecd (127.0.196.194:34129), 0743aa7c73d64ffc8659e88d576805c0 (127.0.196.195:44785)
I20260925 19:05:20.977483 8603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a6134c7393c49dda0906f7b0bc59ab6" candidate_uuid: "df934d2b8daf44219344ce39308d83e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81a2855b68448ad9c7d40da4be26ecd" is_pre_election: true
I20260925 19:05:20.977483 8743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a6134c7393c49dda0906f7b0bc59ab6" candidate_uuid: "df934d2b8daf44219344ce39308d83e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0743aa7c73d64ffc8659e88d576805c0" is_pre_election: true
I20260925 19:05:20.977572 8743 raft_consensus.cc:2454] T 6a6134c7393c49dda0906f7b0bc59ab6 P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df934d2b8daf44219344ce39308d83e5 in term 0.
I20260925 19:05:20.977572 8603 raft_consensus.cc:2454] T 6a6134c7393c49dda0906f7b0bc59ab6 P b81a2855b68448ad9c7d40da4be26ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df934d2b8daf44219344ce39308d83e5 in term 0.
I20260925 19:05:20.977810 8397 leader_election.cc:302] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b81a2855b68448ad9c7d40da4be26ecd, df934d2b8daf44219344ce39308d83e5; no voters:
I20260925 19:05:20.978127 8871 raft_consensus.cc:2781] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:05:20.978178 8871 raft_consensus.cc:482] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:05:20.978209 8871 raft_consensus.cc:3037] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:20.979133 8871 raft_consensus.cc:504] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.979280 8871 leader_election.cc:288] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5 [CANDIDATE]: Term 1 election: Requested vote from peers b81a2855b68448ad9c7d40da4be26ecd (127.0.196.194:34129), 0743aa7c73d64ffc8659e88d576805c0 (127.0.196.195:44785)
I20260925 19:05:20.979506 8603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a6134c7393c49dda0906f7b0bc59ab6" candidate_uuid: "df934d2b8daf44219344ce39308d83e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81a2855b68448ad9c7d40da4be26ecd"
I20260925 19:05:20.979502 8743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a6134c7393c49dda0906f7b0bc59ab6" candidate_uuid: "df934d2b8daf44219344ce39308d83e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0743aa7c73d64ffc8659e88d576805c0"
I20260925 19:05:20.979586 8603 raft_consensus.cc:3037] T 6a6134c7393c49dda0906f7b0bc59ab6 P b81a2855b68448ad9c7d40da4be26ecd [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:20.979592 8743 raft_consensus.cc:3037] T 6a6134c7393c49dda0906f7b0bc59ab6 P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:20.980353 8743 raft_consensus.cc:2454] T 6a6134c7393c49dda0906f7b0bc59ab6 P 0743aa7c73d64ffc8659e88d576805c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df934d2b8daf44219344ce39308d83e5 in term 1.
I20260925 19:05:20.980353 8603 raft_consensus.cc:2454] T 6a6134c7393c49dda0906f7b0bc59ab6 P b81a2855b68448ad9c7d40da4be26ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df934d2b8daf44219344ce39308d83e5 in term 1.
I20260925 19:05:20.980588 8397 leader_election.cc:302] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b81a2855b68448ad9c7d40da4be26ecd, df934d2b8daf44219344ce39308d83e5; no voters:
I20260925 19:05:20.980703 8871 raft_consensus.cc:2781] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:05:20.980774 8871 raft_consensus.cc:686] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5 [term 1 LEADER]: Becoming Leader. State: Replica: df934d2b8daf44219344ce39308d83e5, State: Running, Role: LEADER
I20260925 19:05:20.980832 8871 consensus_queue.cc:237] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:20.981508 8305 catalog_manager.cc:5495] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to df934d2b8daf44219344ce39308d83e5 (127.0.196.193). New cstate: current_term: 1 leader_uuid: "df934d2b8daf44219344ce39308d83e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } health_report { overall_health: UNKNOWN } } }
I20260925 19:05:20.989867 8813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:20.987320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38266 (local address 127.0.196.193:46635)
0925 19:05:20.987379 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:20.987381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:20.987390 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:05:20.987423 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:20.987426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:20.987440 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:20.987505 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:05:20.987656 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.988176 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.989034 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:20.989268 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:20.989379 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:20.989595 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:05:20.989634 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:05:20.989684 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:20.989749 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 19:05:20.989769 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 19:05:20.990722 8743 raft_consensus.cc:1261] T 6a6134c7393c49dda0906f7b0bc59ab6 P 0743aa7c73d64ffc8659e88d576805c0 [term 1 FOLLOWER]: Refusing update from remote peer df934d2b8daf44219344ce39308d83e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:05:20.990732 8603 raft_consensus.cc:1261] T 6a6134c7393c49dda0906f7b0bc59ab6 P b81a2855b68448ad9c7d40da4be26ecd [term 1 FOLLOWER]: Refusing update from remote peer df934d2b8daf44219344ce39308d83e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:05:20.990959 8871 consensus_queue.cc:1035] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:05:20.991087 8825 consensus_queue.cc:1035] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:05:20.995190 8305 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:05:20.995307 8305 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:05:20.996016 8305 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 629f983bc1f14546b301430b0bbe9824: Sending DeleteTablet for 3 replicas of tablet 6a6134c7393c49dda0906f7b0bc59ab6
I20260925 19:05:20.996263 8443 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a6134c7393c49dda0906f7b0bc59ab6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:05:20 UTC) from {username='oryx', principal='oryx/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:38254
I20260925 19:05:20.996363 8840 tablet_replica.cc:330] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5: stopping tablet replica
I20260925 19:05:20.996428 8840 raft_consensus.cc:2229] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:05:20.996457 8583 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a6134c7393c49dda0906f7b0bc59ab6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:05:20 UTC) from {username='oryx', principal='oryx/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:36914
I20260925 19:05:20.996493 8840 raft_consensus.cc:2258] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:05:20.996579 8723 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a6134c7393c49dda0906f7b0bc59ab6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:05:20 UTC) from {username='oryx', principal='oryx/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:60802
I20260925 19:05:20.996682 8841 tablet_replica.cc:330] T 6a6134c7393c49dda0906f7b0bc59ab6 P b81a2855b68448ad9c7d40da4be26ecd: stopping tablet replica
I20260925 19:05:20.996752 8841 raft_consensus.cc:2229] T 6a6134c7393c49dda0906f7b0bc59ab6 P b81a2855b68448ad9c7d40da4be26ecd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:05:20.996848 8841 raft_consensus.cc:2258] T 6a6134c7393c49dda0906f7b0bc59ab6 P b81a2855b68448ad9c7d40da4be26ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:05:20.997018 8842 tablet_replica.cc:330] T 6a6134c7393c49dda0906f7b0bc59ab6 P 0743aa7c73d64ffc8659e88d576805c0: stopping tablet replica
I20260925 19:05:20.997084 8842 raft_consensus.cc:2229] T 6a6134c7393c49dda0906f7b0bc59ab6 P 0743aa7c73d64ffc8659e88d576805c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:05:20.997119 8842 raft_consensus.cc:2258] T 6a6134c7393c49dda0906f7b0bc59ab6 P 0743aa7c73d64ffc8659e88d576805c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:05:20.997454 8840 ts_tablet_manager.cc:1905] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:05:20.998929 8840 ts_tablet_manager.cc:1918] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:05:20.998991 8840 log.cc:1196] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-0/wal/wals/6a6134c7393c49dda0906f7b0bc59ab6
I20260925 19:05:20.999236 8840 ts_tablet_manager.cc:1939] T 6a6134c7393c49dda0906f7b0bc59ab6 P df934d2b8daf44219344ce39308d83e5: Deleting consensus metadata
I20260925 19:05:20.999500 8305 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:05:20.999696 8305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:05:21.000018 8290 catalog_manager.cc:4841] TS df934d2b8daf44219344ce39308d83e5 (127.0.196.193:46635): tablet 6a6134c7393c49dda0906f7b0bc59ab6 (table test-table [id=0a641d375a534f0b8c60f30cc055a572]) successfully deleted
I20260925 19:05:21.000149 8841 ts_tablet_manager.cc:1905] T 6a6134c7393c49dda0906f7b0bc59ab6 P b81a2855b68448ad9c7d40da4be26ecd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:05:21.001408 8583 tablet_service.cc:1467] Processing CreateTablet for tablet 1c3a31a0d1174fc885caf4913174bc98 (DEFAULT_TABLE table=test-table [id=e7650f8aabd54fd283eb926795ce7213]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:05:21.001537 8583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c3a31a0d1174fc885caf4913174bc98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:21.001780 8841 ts_tablet_manager.cc:1918] T 6a6134c7393c49dda0906f7b0bc59ab6 P b81a2855b68448ad9c7d40da4be26ecd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:05:21.001830 8841 log.cc:1196] T 6a6134c7393c49dda0906f7b0bc59ab6 P b81a2855b68448ad9c7d40da4be26ecd: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-1/wal/wals/6a6134c7393c49dda0906f7b0bc59ab6
I20260925 19:05:21.002051 8841 ts_tablet_manager.cc:1939] T 6a6134c7393c49dda0906f7b0bc59ab6 P b81a2855b68448ad9c7d40da4be26ecd: Deleting consensus metadata
I20260925 19:05:21.003067 8723 tablet_service.cc:1467] Processing CreateTablet for tablet 1c3a31a0d1174fc885caf4913174bc98 (DEFAULT_TABLE table=test-table [id=e7650f8aabd54fd283eb926795ce7213]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:05:21.003175 8723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c3a31a0d1174fc885caf4913174bc98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:21.004256 8818 tablet_bootstrap.cc:492] T 1c3a31a0d1174fc885caf4913174bc98 P 0743aa7c73d64ffc8659e88d576805c0: Bootstrap starting.
I20260925 19:05:21.004451 8842 ts_tablet_manager.cc:1905] T 6a6134c7393c49dda0906f7b0bc59ab6 P 0743aa7c73d64ffc8659e88d576805c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:05:21.004595 8818 tablet_bootstrap.cc:654] T 1c3a31a0d1174fc885caf4913174bc98 P 0743aa7c73d64ffc8659e88d576805c0: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:21.005638 8293 catalog_manager.cc:4841] TS b81a2855b68448ad9c7d40da4be26ecd (127.0.196.194:34129): tablet 6a6134c7393c49dda0906f7b0bc59ab6 (table test-table [id=0a641d375a534f0b8c60f30cc055a572]) successfully deleted
I20260925 19:05:21.005941 8443 tablet_service.cc:1467] Processing CreateTablet for tablet 1c3a31a0d1174fc885caf4913174bc98 (DEFAULT_TABLE table=test-table [id=e7650f8aabd54fd283eb926795ce7213]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:05:21.006014 8817 tablet_bootstrap.cc:492] T 1c3a31a0d1174fc885caf4913174bc98 P b81a2855b68448ad9c7d40da4be26ecd: Bootstrap starting.
I20260925 19:05:21.006089 8443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c3a31a0d1174fc885caf4913174bc98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:21.006794 8817 tablet_bootstrap.cc:654] T 1c3a31a0d1174fc885caf4913174bc98 P b81a2855b68448ad9c7d40da4be26ecd: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:21.007675 8817 tablet_bootstrap.cc:492] T 1c3a31a0d1174fc885caf4913174bc98 P b81a2855b68448ad9c7d40da4be26ecd: No bootstrap required, opened a new log
I20260925 19:05:21.007719 8819 tablet_bootstrap.cc:492] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5: Bootstrap starting.
I20260925 19:05:21.007726 8817 ts_tablet_manager.cc:1397] T 1c3a31a0d1174fc885caf4913174bc98 P b81a2855b68448ad9c7d40da4be26ecd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:05:21.007905 8817 raft_consensus.cc:348] T 1c3a31a0d1174fc885caf4913174bc98 P b81a2855b68448ad9c7d40da4be26ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:21.008008 8817 raft_consensus.cc:374] T 1c3a31a0d1174fc885caf4913174bc98 P b81a2855b68448ad9c7d40da4be26ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:21.008029 8817 raft_consensus.cc:729] T 1c3a31a0d1174fc885caf4913174bc98 P b81a2855b68448ad9c7d40da4be26ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81a2855b68448ad9c7d40da4be26ecd, State: Initialized, Role: FOLLOWER
I20260925 19:05:21.008086 8817 consensus_queue.cc:260] T 1c3a31a0d1174fc885caf4913174bc98 P b81a2855b68448ad9c7d40da4be26ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:21.008167 8819 tablet_bootstrap.cc:654] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:21.008212 8817 ts_tablet_manager.cc:1428] T 1c3a31a0d1174fc885caf4913174bc98 P b81a2855b68448ad9c7d40da4be26ecd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:21.008790 8842 ts_tablet_manager.cc:1918] T 6a6134c7393c49dda0906f7b0bc59ab6 P 0743aa7c73d64ffc8659e88d576805c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:05:21.008873 8842 log.cc:1196] T 6a6134c7393c49dda0906f7b0bc59ab6 P 0743aa7c73d64ffc8659e88d576805c0: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-2/wal/wals/6a6134c7393c49dda0906f7b0bc59ab6
I20260925 19:05:21.009124 8842 ts_tablet_manager.cc:1939] T 6a6134c7393c49dda0906f7b0bc59ab6 P 0743aa7c73d64ffc8659e88d576805c0: Deleting consensus metadata
I20260925 19:05:21.009522 8818 tablet_bootstrap.cc:492] T 1c3a31a0d1174fc885caf4913174bc98 P 0743aa7c73d64ffc8659e88d576805c0: No bootstrap required, opened a new log
I20260925 19:05:21.009529 8293 catalog_manager.cc:4841] TS 0743aa7c73d64ffc8659e88d576805c0 (127.0.196.195:44785): tablet 6a6134c7393c49dda0906f7b0bc59ab6 (table test-table [id=0a641d375a534f0b8c60f30cc055a572]) successfully deleted
I20260925 19:05:21.009578 8818 ts_tablet_manager.cc:1397] T 1c3a31a0d1174fc885caf4913174bc98 P 0743aa7c73d64ffc8659e88d576805c0: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260925 19:05:21.009759 8818 raft_consensus.cc:348] T 1c3a31a0d1174fc885caf4913174bc98 P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:21.009848 8818 raft_consensus.cc:374] T 1c3a31a0d1174fc885caf4913174bc98 P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:21.009873 8818 raft_consensus.cc:729] T 1c3a31a0d1174fc885caf4913174bc98 P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0743aa7c73d64ffc8659e88d576805c0, State: Initialized, Role: FOLLOWER
I20260925 19:05:21.009943 8818 consensus_queue.cc:260] T 1c3a31a0d1174fc885caf4913174bc98 P 0743aa7c73d64ffc8659e88d576805c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:21.010003 8819 tablet_bootstrap.cc:492] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5: No bootstrap required, opened a new log
I20260925 19:05:21.010046 8819 ts_tablet_manager.cc:1397] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:05:21.010066 8818 ts_tablet_manager.cc:1428] T 1c3a31a0d1174fc885caf4913174bc98 P 0743aa7c73d64ffc8659e88d576805c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:21.010183 8819 raft_consensus.cc:348] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:21.010265 8819 raft_consensus.cc:374] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:21.010288 8819 raft_consensus.cc:729] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df934d2b8daf44219344ce39308d83e5, State: Initialized, Role: FOLLOWER
I20260925 19:05:21.010344 8819 consensus_queue.cc:260] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:21.010440 8819 ts_tablet_manager.cc:1428] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 19:05:21.217731 8871 raft_consensus.cc:482] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:05:21.217845 8871 raft_consensus.cc:504] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:21.218108 8871 leader_election.cc:288] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b81a2855b68448ad9c7d40da4be26ecd (127.0.196.194:34129), 0743aa7c73d64ffc8659e88d576805c0 (127.0.196.195:44785)
I20260925 19:05:21.218341 8603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c3a31a0d1174fc885caf4913174bc98" candidate_uuid: "df934d2b8daf44219344ce39308d83e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81a2855b68448ad9c7d40da4be26ecd" is_pre_election: true
I20260925 19:05:21.218379 8743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c3a31a0d1174fc885caf4913174bc98" candidate_uuid: "df934d2b8daf44219344ce39308d83e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0743aa7c73d64ffc8659e88d576805c0" is_pre_election: true
I20260925 19:05:21.218451 8603 raft_consensus.cc:2454] T 1c3a31a0d1174fc885caf4913174bc98 P b81a2855b68448ad9c7d40da4be26ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df934d2b8daf44219344ce39308d83e5 in term 0.
I20260925 19:05:21.218463 8743 raft_consensus.cc:2454] T 1c3a31a0d1174fc885caf4913174bc98 P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df934d2b8daf44219344ce39308d83e5 in term 0.
I20260925 19:05:21.218648 8397 leader_election.cc:302] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b81a2855b68448ad9c7d40da4be26ecd, df934d2b8daf44219344ce39308d83e5; no voters:
I20260925 19:05:21.218755 8871 raft_consensus.cc:2781] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:05:21.218828 8871 raft_consensus.cc:482] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:05:21.218860 8871 raft_consensus.cc:3037] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:21.219591 8871 raft_consensus.cc:504] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:21.219724 8871 leader_election.cc:288] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5 [CANDIDATE]: Term 1 election: Requested vote from peers b81a2855b68448ad9c7d40da4be26ecd (127.0.196.194:34129), 0743aa7c73d64ffc8659e88d576805c0 (127.0.196.195:44785)
I20260925 19:05:21.219915 8743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c3a31a0d1174fc885caf4913174bc98" candidate_uuid: "df934d2b8daf44219344ce39308d83e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0743aa7c73d64ffc8659e88d576805c0"
I20260925 19:05:21.219889 8603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c3a31a0d1174fc885caf4913174bc98" candidate_uuid: "df934d2b8daf44219344ce39308d83e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81a2855b68448ad9c7d40da4be26ecd"
I20260925 19:05:21.219998 8743 raft_consensus.cc:3037] T 1c3a31a0d1174fc885caf4913174bc98 P 0743aa7c73d64ffc8659e88d576805c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:21.220011 8603 raft_consensus.cc:3037] T 1c3a31a0d1174fc885caf4913174bc98 P b81a2855b68448ad9c7d40da4be26ecd [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:21.220778 8603 raft_consensus.cc:2454] T 1c3a31a0d1174fc885caf4913174bc98 P b81a2855b68448ad9c7d40da4be26ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df934d2b8daf44219344ce39308d83e5 in term 1.
I20260925 19:05:21.220779 8743 raft_consensus.cc:2454] T 1c3a31a0d1174fc885caf4913174bc98 P 0743aa7c73d64ffc8659e88d576805c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df934d2b8daf44219344ce39308d83e5 in term 1.
I20260925 19:05:21.221029 8397 leader_election.cc:302] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0743aa7c73d64ffc8659e88d576805c0, df934d2b8daf44219344ce39308d83e5; no voters:
I20260925 19:05:21.221262 8871 raft_consensus.cc:2781] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:05:21.221339 8871 raft_consensus.cc:686] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5 [term 1 LEADER]: Becoming Leader. State: Replica: df934d2b8daf44219344ce39308d83e5, State: Running, Role: LEADER
I20260925 19:05:21.221426 8871 consensus_queue.cc:237] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } }
I20260925 19:05:21.222090 8305 catalog_manager.cc:5495] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to df934d2b8daf44219344ce39308d83e5 (127.0.196.193). New cstate: current_term: 1 leader_uuid: "df934d2b8daf44219344ce39308d83e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df934d2b8daf44219344ce39308d83e5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 } health_report { overall_health: UNKNOWN } } }
I20260925 19:05:21.241434 8743 raft_consensus.cc:1261] T 1c3a31a0d1174fc885caf4913174bc98 P 0743aa7c73d64ffc8659e88d576805c0 [term 1 FOLLOWER]: Refusing update from remote peer df934d2b8daf44219344ce39308d83e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:05:21.241417 8603 raft_consensus.cc:1261] T 1c3a31a0d1174fc885caf4913174bc98 P b81a2855b68448ad9c7d40da4be26ecd [term 1 FOLLOWER]: Refusing update from remote peer df934d2b8daf44219344ce39308d83e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:05:21.241647 8871 consensus_queue.cc:1035] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0743aa7c73d64ffc8659e88d576805c0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:05:21.241794 8848 consensus_queue.cc:1035] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81a2855b68448ad9c7d40da4be26ecd" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:05:21.251729 8306 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:05:21.251830 8306 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:05:21.252524 8306 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 629f983bc1f14546b301430b0bbe9824: Sending DeleteTablet for 3 replicas of tablet 1c3a31a0d1174fc885caf4913174bc98
I20260925 19:05:21.252799 8443 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c3a31a0d1174fc885caf4913174bc98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:05:21 UTC) from {username='oryx', principal='oryx/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:38254
I20260925 19:05:21.252873 8583 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c3a31a0d1174fc885caf4913174bc98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:05:21 UTC) from {username='oryx', principal='oryx/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:36914
I20260925 19:05:21.252851 8723 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c3a31a0d1174fc885caf4913174bc98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:05:21 UTC) from {username='oryx', principal='oryx/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:60802
I20260925 19:05:21.252985 8842 tablet_replica.cc:330] T 1c3a31a0d1174fc885caf4913174bc98 P 0743aa7c73d64ffc8659e88d576805c0: stopping tablet replica
I20260925 19:05:21.252988 8840 tablet_replica.cc:330] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5: stopping tablet replica
I20260925 19:05:21.253049 8842 raft_consensus.cc:2229] T 1c3a31a0d1174fc885caf4913174bc98 P 0743aa7c73d64ffc8659e88d576805c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:05:21.253049 8840 raft_consensus.cc:2229] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:05:21.253095 8842 raft_consensus.cc:2258] T 1c3a31a0d1174fc885caf4913174bc98 P 0743aa7c73d64ffc8659e88d576805c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:05:21.253129 8840 raft_consensus.cc:2258] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:05:21.253221 8841 tablet_replica.cc:330] T 1c3a31a0d1174fc885caf4913174bc98 P b81a2855b68448ad9c7d40da4be26ecd: stopping tablet replica
I20260925 19:05:21.253289 8841 raft_consensus.cc:2229] T 1c3a31a0d1174fc885caf4913174bc98 P b81a2855b68448ad9c7d40da4be26ecd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:05:21.253336 8841 raft_consensus.cc:2258] T 1c3a31a0d1174fc885caf4913174bc98 P b81a2855b68448ad9c7d40da4be26ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:05:21.253645 8840 ts_tablet_manager.cc:1905] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:05:21.253993 8841 ts_tablet_manager.cc:1905] T 1c3a31a0d1174fc885caf4913174bc98 P b81a2855b68448ad9c7d40da4be26ecd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:05:21.255527 8840 ts_tablet_manager.cc:1918] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:05:21.255601 8840 log.cc:1196] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-0/wal/wals/1c3a31a0d1174fc885caf4913174bc98
I20260925 19:05:21.255707 8841 ts_tablet_manager.cc:1918] T 1c3a31a0d1174fc885caf4913174bc98 P b81a2855b68448ad9c7d40da4be26ecd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:05:21.255771 8841 log.cc:1196] T 1c3a31a0d1174fc885caf4913174bc98 P b81a2855b68448ad9c7d40da4be26ecd: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-1/wal/wals/1c3a31a0d1174fc885caf4913174bc98
I20260925 19:05:21.255878 8840 ts_tablet_manager.cc:1939] T 1c3a31a0d1174fc885caf4913174bc98 P df934d2b8daf44219344ce39308d83e5: Deleting consensus metadata
I20260925 19:05:21.256043 8841 ts_tablet_manager.cc:1939] T 1c3a31a0d1174fc885caf4913174bc98 P b81a2855b68448ad9c7d40da4be26ecd: Deleting consensus metadata
I20260925 19:05:21.256179 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 8378
I20260925 19:05:21.256763 8290 catalog_manager.cc:4841] TS df934d2b8daf44219344ce39308d83e5 (127.0.196.193:46635): tablet 1c3a31a0d1174fc885caf4913174bc98 (table test-table [id=e7650f8aabd54fd283eb926795ce7213]) successfully deleted
I20260925 19:05:21.257364 8293 catalog_manager.cc:4841] TS b81a2855b68448ad9c7d40da4be26ecd (127.0.196.194:34129): tablet 1c3a31a0d1174fc885caf4913174bc98 (table test-table [id=e7650f8aabd54fd283eb926795ce7213]) successfully deleted
I20260925 19:05:21.258188 8842 ts_tablet_manager.cc:1905] T 1c3a31a0d1174fc885caf4913174bc98 P 0743aa7c73d64ffc8659e88d576805c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:05:21.259582 8842 ts_tablet_manager.cc:1918] T 1c3a31a0d1174fc885caf4913174bc98 P 0743aa7c73d64ffc8659e88d576805c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:05:21.259636 8842 log.cc:1196] T 1c3a31a0d1174fc885caf4913174bc98 P 0743aa7c73d64ffc8659e88d576805c0: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790363087086945-787-0/minicluster-data/ts-2/wal/wals/1c3a31a0d1174fc885caf4913174bc98
I20260925 19:05:21.259913 8842 ts_tablet_manager.cc:1939] T 1c3a31a0d1174fc885caf4913174bc98 P 0743aa7c73d64ffc8659e88d576805c0: Deleting consensus metadata
I20260925 19:05:21.260265 8293 catalog_manager.cc:4841] TS 0743aa7c73d64ffc8659e88d576805c0 (127.0.196.195:44785): tablet 1c3a31a0d1174fc885caf4913174bc98 (table test-table [id=e7650f8aabd54fd283eb926795ce7213]) successfully deleted
I20260925 19:05:21.264267 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 8518
I20260925 19:05:21.270519 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 8658
I20260925 19:05:21.277439 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 8274
2026-09-25T19:05:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2546 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:05:21 dist-test-slave-rn98 krb5kdc[8901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:05:21 dist-test-slave-rn98 krb5kdc[8901](info): set up 2 sockets
Sep 25 19:05:21 dist-test-slave-rn98 krb5kdc[8901](info): commencing operation
krb5kdc: starting...
W20260925 19:05:22.323416 787 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:05:22 dist-test-slave-rn98 krb5kdc[8901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:05:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:05:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:05:22.450605 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:33513
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:43491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:33513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:22.533776 8914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:22.534008 8914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:22.534034 8914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:22.535220 8914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:05:22.535300 8914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:22.535317 8914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:22.535331 8914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:05:22.535342 8914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:05:22.536865 8914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:43491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:33513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:33513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:22.537074 8914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:22.537249 8914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:22.542268 8920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:22.542270 8922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:22.542304 8919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:22.542577 8914 server_base.cc:1034] running on GCE node
I20260925 19:05:22.542847 8914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:22.543149 8914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:22.544333 8914 hybrid_clock.cc:648] HybridClock initialized: now 1790363122544286 us; error 66 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T19:05:22Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1269 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:05:22 dist-test-slave-rn98 krb5kdc[8927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:05:22 dist-test-slave-rn98 krb5kdc[8927](info): set up 2 sockets
Sep 25 19:05:22 dist-test-slave-rn98 krb5kdc[8927](info): commencing operation
krb5kdc: starting...
W20260925 19:05:23.595463 787 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:05:23 dist-test-slave-rn98 krb5kdc[8927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:05:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:05:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:05:23.721467 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:34733
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:34733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:23.803889 8940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:23.804123 8940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:23.804145 8940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:23.805316 8940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:05:23.805362 8940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:23.805377 8940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:23.805390 8940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:05:23.805403 8940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:05:23.807009 8940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:34733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:34733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:23.807236 8940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:23.807411 8940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:23.812217 8948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:23.812220 8946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:23.812467 8940 server_base.cc:1034] running on GCE node
W20260925 19:05:23.812258 8945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:23.812741 8940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:23.813072 8940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:23.814216 8940 hybrid_clock.cc:648] HybridClock initialized: now 1790363123814192 us; error 37 us; skew 500 ppm
Sep 25 19:05:23 dist-test-slave-rn98 krb5kdc[8927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:23.819748 8940 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:05:23.820216 8940 webserver.cc:460] Webserver started at http://127.0.196.254:43037/ using document root <none> and password file <none>
I20260925 19:05:23.820415 8940 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:23.820456 8940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:23.820554 8940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:23.821574 8940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "4e3b4361fa4e4de68745c654812c2364"
format_stamp: "Formatted at 2026-09-25 19:05:23 on dist-test-slave-rn98"
server_key: "d97df262653c92771fb786dc8eef0450"
server_key_iv: "685b0c26296954f0ee2fa47192cd2f6e"
server_key_version: "encryptionkey@0"
I20260925 19:05:23.821878 8940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "4e3b4361fa4e4de68745c654812c2364"
format_stamp: "Formatted at 2026-09-25 19:05:23 on dist-test-slave-rn98"
server_key: "d97df262653c92771fb786dc8eef0450"
server_key_iv: "685b0c26296954f0ee2fa47192cd2f6e"
server_key_version: "encryptionkey@0"
I20260925 19:05:23.823395 8940 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:05:23.824220 8955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:23.824450 8940 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:23.824573 8940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "4e3b4361fa4e4de68745c654812c2364"
format_stamp: "Formatted at 2026-09-25 19:05:23 on dist-test-slave-rn98"
server_key: "d97df262653c92771fb786dc8eef0450"
server_key_iv: "685b0c26296954f0ee2fa47192cd2f6e"
server_key_version: "encryptionkey@0"
I20260925 19:05:23.824656 8940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:23.837666 8940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:23.839900 8940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:23.840071 8940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:23.844496 8940 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:34733
I20260925 19:05:23.844540 9007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:34733 every 8 connection(s)
I20260925 19:05:23.845064 8940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:05:23.845734 9008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:23.846527 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 8940
I20260925 19:05:23.846781 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:05:23.846964 787 external_mini_cluster.cc:1428] Setting key f357d8484f16b85d359dacf6a4c52e7a
I20260925 19:05:23.849090 9008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364: Bootstrap starting.
I20260925 19:05:23.849813 9008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:23.850119 9008 log.cc:824] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:23.850841 9008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364: No bootstrap required, opened a new log
I20260925 19:05:23.853163 9008 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3b4361fa4e4de68745c654812c2364" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 34733 } }
I20260925 19:05:23.853297 9008 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:23.853319 9008 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e3b4361fa4e4de68745c654812c2364, State: Initialized, Role: FOLLOWER
I20260925 19:05:23.853461 9008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3b4361fa4e4de68745c654812c2364" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 34733 } }
I20260925 19:05:23.853541 9008 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:05:23.853577 9008 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:05:23.853626 9008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 19:05:23 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:05:23.854235 9008 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3b4361fa4e4de68745c654812c2364" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 34733 } }
I20260925 19:05:23.854379 9008 leader_election.cc:302] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e3b4361fa4e4de68745c654812c2364; no voters:
I20260925 19:05:23.854571 9008 leader_election.cc:288] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:05:23.854604 9013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:05:23.854736 9013 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [term 1 LEADER]: Becoming Leader. State: Replica: 4e3b4361fa4e4de68745c654812c2364, State: Running, Role: LEADER
I20260925 19:05:23.854828 9013 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3b4361fa4e4de68745c654812c2364" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 34733 } }
I20260925 19:05:23.854889 9008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:05:23.855522 9013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e3b4361fa4e4de68745c654812c2364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3b4361fa4e4de68745c654812c2364" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 34733 } } }
I20260925 19:05:23.855906 9013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:23.856048 9013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e3b4361fa4e4de68745c654812c2364. Latest consensus state: current_term: 1 leader_uuid: "4e3b4361fa4e4de68745c654812c2364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3b4361fa4e4de68745c654812c2364" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 34733 } } }
I20260925 19:05:23.856120 9013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [sys.catalog]: This master's current role is: LEADER
W20260925 19:05:23.856935 9028 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:05:23.856993 9028 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:05:23.857105 9024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:05:23.857275 9024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:05:23.858533 9011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:23.847534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49622 (local address 127.0.196.254:34733)
0925 19:05:23.847725 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:23.847735 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:23.847769 (+ 34us) server_negotiation.cc:407] Connection header received
0925 19:05:23.848295 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:23.848305 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:23.848383 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:23.848539 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:23.849129 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:23.850478 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:23.851356 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:23.851982 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:23.852107 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:23.854544 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:23.854548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:23.854548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:23.854567 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:23.856772 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:23.857235 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:23.857240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:23.857240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:23.857360 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:23.857584 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:23.857585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:23.857585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:23.857897 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:23.858048 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:23.858181 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 19:05:23.858338 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":80,"threads_started":1}
I20260925 19:05:23.859215 9024 catalog_manager.cc:1347] Generated new cluster ID: e2af2f58d01744038acbce7f3f78d775
I20260925 19:05:23.859266 9024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:05:23.870398 9024 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:05:23.871114 9024 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:05:23.881253 9024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364: Generated new TSK 0
I20260925 19:05:23.881491 9024 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
I20260925 19:05:23.918556 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:0
--local_ip_for_outbound_sockets=127.0.196.193
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:34733
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:24.028158 9034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:24.028381 9034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:24.028405 9034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:24.029603 9034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:24.029650 9034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:24.029678 9034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:05:24.031173 9034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:34733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:24.031376 9034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:24.031569 9034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:05:24.036863 9034 server_base.cc:1034] running on GCE node
W20260925 19:05:24.036819 9042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:24.036819 9039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:24.036823 9040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:24.037400 9034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:24.037679 9034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:24.038841 9034 hybrid_clock.cc:648] HybridClock initialized: now 1790363124038841 us; error 41 us; skew 500 ppm
Sep 25 19:05:24 dist-test-slave-rn98 krb5kdc[8927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:24.041435 9034 init.cc:377] Logged in from keytab as kudu/127.0.196.193@KRBTEST.COM (short username kudu)
I20260925 19:05:24.041932 9034 webserver.cc:460] Webserver started at http://127.0.196.193:42121/ using document root <none> and password file <none>
I20260925 19:05:24.042199 9034 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:24.042272 9034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:24.042416 9034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:24.043525 9034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/data/instance:
uuid: "40b81bd96db64922942011cba782f76f"
format_stamp: "Formatted at 2026-09-25 19:05:24 on dist-test-slave-rn98"
server_key: "8776f02d4a6fee39c0832378bb084f52"
server_key_iv: "5760afa31e3ffc381aa9625a51d1863c"
server_key_version: "encryptionkey@0"
I20260925 19:05:24.043885 9034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/wal/instance:
uuid: "40b81bd96db64922942011cba782f76f"
format_stamp: "Formatted at 2026-09-25 19:05:24 on dist-test-slave-rn98"
server_key: "8776f02d4a6fee39c0832378bb084f52"
server_key_iv: "5760afa31e3ffc381aa9625a51d1863c"
server_key_version: "encryptionkey@0"
I20260925 19:05:24.045244 9034 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:05:24.046084 9049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:24.046306 9034 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:24.046432 9034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "40b81bd96db64922942011cba782f76f"
format_stamp: "Formatted at 2026-09-25 19:05:24 on dist-test-slave-rn98"
server_key: "8776f02d4a6fee39c0832378bb084f52"
server_key_iv: "5760afa31e3ffc381aa9625a51d1863c"
server_key_version: "encryptionkey@0"
I20260925 19:05:24.046522 9034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:24.069672 9034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:24.072208 9034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:24.072404 9034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:24.072700 9034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:24.073050 9034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:24.073117 9034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:24.073163 9034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:24.073194 9034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:24.079752 9034 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:40545
I20260925 19:05:24.079799 9162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:40545 every 8 connection(s)
I20260925 19:05:24.080171 9034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
I20260925 19:05:24.085140 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 9034
I20260925 19:05:24.085281 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/wal/instance
I20260925 19:05:24.085484 787 external_mini_cluster.cc:1428] Setting key ad5cda076045c413eaa9095291226578
Sep 25 19:05:24 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:05:24.091135 9011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:24.081007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:44427 (local address 127.0.196.254:34733)
0925 19:05:24.081080 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:24.081082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:24.081304 (+ 222us) server_negotiation.cc:407] Connection header received
0925 19:05:24.082384 (+ 1080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:24.082385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:24.082397 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:24.082461 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:24.083236 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.083793 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.084942 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.085149 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.085216 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:24.087900 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:24.087902 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:24.087903 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:24.087912 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:24.089500 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:24.089992 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:24.089994 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:24.089994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:24.090057 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:24.090346 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:24.090347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:24.090347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:24.090499 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:24.090581 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:24.090927 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 19:05:24.091032 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 19:05:24.091332 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:24.081002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.193:44427)
0925 19:05:24.081163 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:24.081204 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:24.082215 (+ 1011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:24.082567 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:24.082571 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:24.082898 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:24.083130 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:24.083135 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.083902 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.083903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:24.084795 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:24.084798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.085255 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.085256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:24.085443 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:24.086168 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:24.086180 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:24.087788 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:24.089612 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:24.089621 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:24.089624 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:24.089881 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:24.090128 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:24.090129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:24.090129 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:24.090274 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:24.090570 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:24.090578 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:24.090777 (+ 199us) client_negotiation.cc:769] Sending connection context
0925 19:05:24.090894 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 19:05:24.091079 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260925 19:05:24.091814 9163 heartbeater.cc:347] Connected to a master server at 127.0.196.254:34733
I20260925 19:05:24.091995 9163 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:24.092218 9163 heartbeater.cc:511] Master 127.0.196.254:34733 requested a full tablet report, sending...
I20260925 19:05:24.092846 8972 ts_manager.cc:194] Registered new tserver with Master: 40b81bd96db64922942011cba782f76f (127.0.196.193:40545)
WARNING: no policy specified for kudu/127.0.196.194@KRBTEST.COM; defaulting to no policy
I20260925 19:05:24.093835 8972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.193@KRBTEST.COM'} at 127.0.196.193:44427
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
I20260925 19:05:24.115038 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:0
--local_ip_for_outbound_sockets=127.0.196.194
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:34733
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:24.199270 9168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:24.199486 9168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:24.199510 9168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:24.200608 9168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:24.200646 9168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:24.200672 9168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:05:24.202099 9168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:34733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:24.202294 9168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:24.202487 9168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:24.208015 9174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:24.208035 9173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:24.208053 9168 server_base.cc:1034] running on GCE node
W20260925 19:05:24.208035 9176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:24.208446 9168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:24.208703 9168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:24.209875 9168 hybrid_clock.cc:648] HybridClock initialized: now 1790363124209860 us; error 53 us; skew 500 ppm
Sep 25 19:05:24 dist-test-slave-rn98 krb5kdc[8927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:24.212648 9168 init.cc:377] Logged in from keytab as kudu/127.0.196.194@KRBTEST.COM (short username kudu)
I20260925 19:05:24.213174 9168 webserver.cc:460] Webserver started at http://127.0.196.194:42841/ using document root <none> and password file <none>
I20260925 19:05:24.213420 9168 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:24.213487 9168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:24.213636 9168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:24.214771 9168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/data/instance:
uuid: "a4ca9804bd4549a38ad9c4239d76a1d5"
format_stamp: "Formatted at 2026-09-25 19:05:24 on dist-test-slave-rn98"
server_key: "b542992d18a03b8edf80c8d268f5fb6a"
server_key_iv: "db1704adfd3ff0550c8af8f0b2e1be09"
server_key_version: "encryptionkey@0"
I20260925 19:05:24.215128 9168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/wal/instance:
uuid: "a4ca9804bd4549a38ad9c4239d76a1d5"
format_stamp: "Formatted at 2026-09-25 19:05:24 on dist-test-slave-rn98"
server_key: "b542992d18a03b8edf80c8d268f5fb6a"
server_key_iv: "db1704adfd3ff0550c8af8f0b2e1be09"
server_key_version: "encryptionkey@0"
I20260925 19:05:24.216539 9168 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:05:24.217237 9183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:24.217403 9168 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:05:24.217506 9168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "a4ca9804bd4549a38ad9c4239d76a1d5"
format_stamp: "Formatted at 2026-09-25 19:05:24 on dist-test-slave-rn98"
server_key: "b542992d18a03b8edf80c8d268f5fb6a"
server_key_iv: "db1704adfd3ff0550c8af8f0b2e1be09"
server_key_version: "encryptionkey@0"
I20260925 19:05:24.217582 9168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:24.235647 9168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:24.237819 9168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:24.237999 9168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:24.238253 9168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:24.238569 9168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:24.238623 9168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:24.238703 9168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:24.238736 9168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:24.244724 9168 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:36427
I20260925 19:05:24.244835 9296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:36427 every 8 connection(s)
I20260925 19:05:24.245147 9168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
I20260925 19:05:24.250298 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 9168
I20260925 19:05:24.250448 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/wal/instance
I20260925 19:05:24.250723 787 external_mini_cluster.cc:1428] Setting key 9f68b307328a11a4f5aae2f842dfd140
Sep 25 19:05:24 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:05:24.256345 9011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:24.245972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:41713 (local address 127.0.196.254:34733)
0925 19:05:24.246047 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:24.246049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:24.246269 (+ 220us) server_negotiation.cc:407] Connection header received
0925 19:05:24.247433 (+ 1164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:24.247434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:24.247444 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:24.247500 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:24.248171 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.248654 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.249803 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.249986 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.250047 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:24.252902 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:24.252904 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:24.252904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:24.252914 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:24.254566 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:24.255153 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:24.255154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:24.255155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:24.255216 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:24.255596 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:24.255597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:24.255597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:24.255705 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:24.255759 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:24.256162 (+ 403us) server_negotiation.cc:299] Negotiation successful
0925 19:05:24.256234 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 19:05:24.256742 9299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:24.245963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.194:41713)
0925 19:05:24.246134 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:24.246173 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:24.247326 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:24.247584 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:24.247588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:24.247875 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:24.248095 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:24.248099 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.248755 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.248756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:24.249710 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:24.249713 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.250129 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.250130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:24.250326 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:24.251013 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:24.251024 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:24.252765 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:24.254687 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:24.254696 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:24.254699 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:24.255071 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:24.255340 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:24.255342 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:24.255342 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:24.255520 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:24.255776 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:24.255782 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:24.256000 (+ 218us) client_negotiation.cc:769] Sending connection context
0925 19:05:24.256135 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 19:05:24.256369 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260925 19:05:24.257272 9297 heartbeater.cc:347] Connected to a master server at 127.0.196.254:34733
I20260925 19:05:24.257409 9297 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:24.257648 9297 heartbeater.cc:511] Master 127.0.196.254:34733 requested a full tablet report, sending...
I20260925 19:05:24.258086 8972 ts_manager.cc:194] Registered new tserver with Master: a4ca9804bd4549a38ad9c4239d76a1d5 (127.0.196.194:36427)
I20260925 19:05:24.258466 8972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.194@KRBTEST.COM'} at 127.0.196.194:41713
WARNING: no policy specified for kudu/127.0.196.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
I20260925 19:05:24.281554 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:0
--local_ip_for_outbound_sockets=127.0.196.195
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:34733
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:24.365485 9302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:24.365715 9302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:24.365741 9302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:24.367115 9302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:24.367168 9302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:24.367198 9302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:05:24.368875 9302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:34733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:24.369151 9302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:24.369410 9302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:24.374830 9308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:24.374846 9310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:24.374830 9307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:24.375134 9302 server_base.cc:1034] running on GCE node
I20260925 19:05:24.375367 9302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:24.375630 9302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:24.376792 9302 hybrid_clock.cc:648] HybridClock initialized: now 1790363124376772 us; error 50 us; skew 500 ppm
Sep 25 19:05:24 dist-test-slave-rn98 krb5kdc[8927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:24.379452 9302 init.cc:377] Logged in from keytab as kudu/127.0.196.195@KRBTEST.COM (short username kudu)
I20260925 19:05:24.379930 9302 webserver.cc:460] Webserver started at http://127.0.196.195:46853/ using document root <none> and password file <none>
I20260925 19:05:24.380182 9302 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:24.380251 9302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:24.380404 9302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:24.381461 9302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/data/instance:
uuid: "4bd8b3cadbbb46f3ad02b5c695bcaae6"
format_stamp: "Formatted at 2026-09-25 19:05:24 on dist-test-slave-rn98"
server_key: "963be4367eb6ac2507b17c17f632241c"
server_key_iv: "594530f73e9dec123f6dea2ecc7cd7f1"
server_key_version: "encryptionkey@0"
I20260925 19:05:24.381865 9302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/wal/instance:
uuid: "4bd8b3cadbbb46f3ad02b5c695bcaae6"
format_stamp: "Formatted at 2026-09-25 19:05:24 on dist-test-slave-rn98"
server_key: "963be4367eb6ac2507b17c17f632241c"
server_key_iv: "594530f73e9dec123f6dea2ecc7cd7f1"
server_key_version: "encryptionkey@0"
I20260925 19:05:24.383313 9302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:05:24.384102 9317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:24.384298 9302 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:24.384399 9302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "4bd8b3cadbbb46f3ad02b5c695bcaae6"
format_stamp: "Formatted at 2026-09-25 19:05:24 on dist-test-slave-rn98"
server_key: "963be4367eb6ac2507b17c17f632241c"
server_key_iv: "594530f73e9dec123f6dea2ecc7cd7f1"
server_key_version: "encryptionkey@0"
I20260925 19:05:24.384531 9302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:24.413411 9302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:24.415725 9302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:24.415917 9302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:24.416180 9302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:24.416488 9302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:24.416551 9302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:24.416594 9302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:24.416617 9302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:24.422777 9302 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:34511
I20260925 19:05:24.422817 9430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:34511 every 8 connection(s)
I20260925 19:05:24.423169 9302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
I20260925 19:05:24.427069 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 9302
I20260925 19:05:24.427196 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/wal/instance
I20260925 19:05:24.427367 787 external_mini_cluster.cc:1428] Setting key bc11ce1c549c860f2d9b563ddc180e36
Sep 25 19:05:24 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:05:24.433732 9011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:24.423998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:60083 (local address 127.0.196.254:34733)
0925 19:05:24.424043 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:24.424045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:24.424328 (+ 283us) server_negotiation.cc:407] Connection header received
0925 19:05:24.425319 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:24.425320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:24.425333 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:24.425373 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:24.426050 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.426720 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.428101 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.428444 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.428499 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:24.430680 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:24.430682 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:24.430682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:24.430690 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:24.432207 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:24.432642 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:24.432643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:24.432643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:24.432703 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:24.433047 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:24.433048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:24.433048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:24.433196 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:24.433263 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:24.433594 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 19:05:24.433664 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:05:24.433925 9433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:24.424032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.195:60083)
0925 19:05:24.424198 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:24.424237 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:24.425190 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:24.425464 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:24.425468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:24.425733 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:24.425929 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:24.425933 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.426813 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.426814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:24.427963 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:24.427982 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.428522 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.428523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:24.428631 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:24.429096 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:24.429104 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:24.430519 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:24.432299 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:24.432305 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:24.432307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:24.432551 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:24.432770 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:24.432771 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:24.432771 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:24.432957 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:24.433293 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:24.433300 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:24.433466 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 19:05:24.433565 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 19:05:24.433696 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20260925 19:05:24.434513 9431 heartbeater.cc:347] Connected to a master server at 127.0.196.254:34733
I20260925 19:05:24.434667 9431 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:24.434872 9431 heartbeater.cc:511] Master 127.0.196.254:34733 requested a full tablet report, sending...
I20260925 19:05:24.435293 8972 ts_manager.cc:194] Registered new tserver with Master: 4bd8b3cadbbb46f3ad02b5c695bcaae6 (127.0.196.195:34511)
I20260925 19:05:24.435699 8972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.195@KRBTEST.COM'} at 127.0.196.195:60083
I20260925 19:05:24.439333 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/oryx_127.0.196.193.keytab.
Entry for principal oryx/127.0.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/oryx_127.0.196.193.keytab.
WARNING: no policy specified for oryx/127.0.196.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/oryx_127.0.196.194.keytab.
Entry for principal oryx/127.0.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/oryx_127.0.196.194.keytab.
WARNING: no policy specified for oryx/127.0.196.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/oryx_127.0.196.195.keytab.
Entry for principal oryx/127.0.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/oryx_127.0.196.195.keytab.
I20260925 19:05:24.518416 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 9034
I20260925 19:05:24.523830 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 9168
I20260925 19:05:24.528769 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 9302
I20260925 19:05:24.533340 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 8940
I20260925 19:05:24.538434 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:34733
--webserver_interface=127.0.196.254
--webserver_port=43037
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:34733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:24.622507 9440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:24.622772 9440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:24.622797 9440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:24.624143 9440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:05:24.624198 9440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:24.624213 9440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:24.624226 9440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:05:24.624238 9440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:05:24.625864 9440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:34733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:34733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=43037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:24.626169 9440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:24.626425 9440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:24.631395 9448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:24.631407 9446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:24.631635 9440 server_base.cc:1034] running on GCE node
W20260925 19:05:24.631405 9445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:24.631954 9440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:24.632287 9440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:24.633491 9440 hybrid_clock.cc:648] HybridClock initialized: now 1790363124633469 us; error 42 us; skew 500 ppm
Sep 25 19:05:24 dist-test-slave-rn98 krb5kdc[8927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:24.636395 9440 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:05:24.636889 9440 webserver.cc:460] Webserver started at http://127.0.196.254:43037/ using document root <none> and password file <none>
I20260925 19:05:24.637141 9440 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:24.637229 9440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:24.638765 9440 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:24.639613 9455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:24.639884 9440 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:24.640004 9440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "4e3b4361fa4e4de68745c654812c2364"
format_stamp: "Formatted at 2026-09-25 19:05:23 on dist-test-slave-rn98"
server_key: "d97df262653c92771fb786dc8eef0450"
server_key_iv: "685b0c26296954f0ee2fa47192cd2f6e"
server_key_version: "encryptionkey@0"
I20260925 19:05:24.640378 9440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:24.650003 9440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:24.652486 9440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:24.652678 9440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:24.656893 9440 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:34733
I20260925 19:05:24.656970 9507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:34733 every 8 connection(s)
I20260925 19:05:24.657558 9440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:05:24.658981 9508 sys_catalog.cc:263] Verifying existing consensus state
I20260925 19:05:24.659680 9508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364: Bootstrap starting.
I20260925 19:05:24.664201 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 9440
I20260925 19:05:24.664537 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:40545
--local_ip_for_outbound_sockets=127.0.196.193
--tserver_master_addrs=127.0.196.254:34733
--webserver_port=42121
--webserver_interface=127.0.196.193
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.196.193
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/oryx_127.0.196.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:05:24.668459 9508 log.cc:824] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:24.670274 9508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:05:24.670608 9508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364: Bootstrap complete.
I20260925 19:05:24.673010 9508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3b4361fa4e4de68745c654812c2364" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 34733 } }
I20260925 19:05:24.673286 9508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e3b4361fa4e4de68745c654812c2364, State: Initialized, Role: FOLLOWER
I20260925 19:05:24.673492 9508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3b4361fa4e4de68745c654812c2364" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 34733 } }
I20260925 19:05:24.673591 9508 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:05:24.673648 9508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:05:24.673696 9508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [term 1 FOLLOWER]: Advancing to term 2
I20260925 19:05:24.674826 9508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3b4361fa4e4de68745c654812c2364" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 34733 } }
I20260925 19:05:24.674949 9508 leader_election.cc:302] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e3b4361fa4e4de68745c654812c2364; no voters:
I20260925 19:05:24.675153 9508 leader_election.cc:288] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 19:05:24.675191 9512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 19:05:24.675297 9512 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [term 2 LEADER]: Becoming Leader. State: Replica: 4e3b4361fa4e4de68745c654812c2364, State: Running, Role: LEADER
I20260925 19:05:24.675463 9508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:05:24.675462 9512 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3b4361fa4e4de68745c654812c2364" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 34733 } }
I20260925 19:05:24.675957 9514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4e3b4361fa4e4de68745c654812c2364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3b4361fa4e4de68745c654812c2364" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 34733 } } }
I20260925 19:05:24.675973 9513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e3b4361fa4e4de68745c654812c2364. Latest consensus state: current_term: 2 leader_uuid: "4e3b4361fa4e4de68745c654812c2364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3b4361fa4e4de68745c654812c2364" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 34733 } } }
I20260925 19:05:24.676069 9514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:24.676090 9513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:24.676493 9520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:05:24.677052 9520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:05:24.677398 9520 catalog_manager.cc:1259] Loaded cluster ID: e2af2f58d01744038acbce7f3f78d775
I20260925 19:05:24.677438 9520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:05:24.678728 9520 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:05:24.678925 9520 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4e3b4361fa4e4de68745c654812c2364: Loaded TSK: 0
I20260925 19:05:24.679144 9520 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 19:05:24.752192 9510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:24.752424 9510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:24.752449 9510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:24.753691 9510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:24.753741 9510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:24.753769 9510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:05:24.755338 9510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/oryx_127.0.196.193.keytab
--principal=oryx/127.0.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:40545
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=42121
--tserver_master_addrs=127.0.196.254:34733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:24.755599 9510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:24.755887 9510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:24.761461 9537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:24.761423 9535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:24.761430 9534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:24.761904 9510 server_base.cc:1034] running on GCE node
I20260925 19:05:24.762104 9510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:24.762382 9510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:24.763545 9510 hybrid_clock.cc:648] HybridClock initialized: now 1790363124763525 us; error 35 us; skew 500 ppm
Sep 25 19:05:24 dist-test-slave-rn98 krb5kdc[8927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363124, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:24.766178 9510 init.cc:377] Logged in from keytab as oryx/127.0.196.193@KRBTEST.COM (short username oryx)
I20260925 19:05:24.766768 9510 webserver.cc:460] Webserver started at http://127.0.196.193:42121/ using document root <none> and password file <none>
I20260925 19:05:24.767040 9510 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:24.767096 9510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:24.768580 9510 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:05:24.769273 9544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:24.769433 9510 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:24.769524 9510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "40b81bd96db64922942011cba782f76f"
format_stamp: "Formatted at 2026-09-25 19:05:24 on dist-test-slave-rn98"
server_key: "8776f02d4a6fee39c0832378bb084f52"
server_key_iv: "5760afa31e3ffc381aa9625a51d1863c"
server_key_version: "encryptionkey@0"
I20260925 19:05:24.769901 9510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:24.796396 9510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:24.798976 9510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:24.799157 9510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:24.799434 9510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:24.799808 9510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:24.799868 9510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:24.799911 9510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:24.799938 9510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:24.807113 9510 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:40545
I20260925 19:05:24.807132 9657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:40545 every 8 connection(s)
I20260925 19:05:24.807539 9510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
I20260925 19:05:24.810245 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 9510
I20260925 19:05:24.810647 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:36427
--local_ip_for_outbound_sockets=127.0.196.194
--tserver_master_addrs=127.0.196.254:34733
--webserver_port=42841
--webserver_interface=127.0.196.194
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.196.194
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/oryx_127.0.196.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 19:05:24 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:24 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 19:05:24.818224 9661 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:24.808442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:52469 (local address 127.0.196.254:34733)
0925 19:05:24.808631 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:24.808640 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:24.808680 (+ 40us) server_negotiation.cc:407] Connection header received
0925 19:05:24.809297 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:24.809302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:24.809355 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:24.809496 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:24.810544 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.812082 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.813724 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.814142 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.814277 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:24.818092 (+ 3815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.193:52469: BlockingRecv error: recv got EOF from 127.0.196.193:52469 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260925 19:05:24.818220 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:24.808309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.193:52469)
0925 19:05:24.808469 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:24.808514 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:24.809060 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:24.809542 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:24.809546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:24.810164 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:24.810401 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:24.810407 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.812200 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.812202 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:24.813627 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:24.813630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.814269 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.814271 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:24.814445 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:24.815090 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:24.815102 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:24.817959 (+ 2857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
W20260925 19:05:24.818539 9658 heartbeater.cc:644] Failed to heartbeat to 127.0.196.254:34733 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.196.254:34733: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Sep 25 19:05:24 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:24 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:24.823843 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:24.818812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.193:51577)
0925 19:05:24.818856 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:24.818870 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:24.818921 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:24.819055 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:24.819056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:24.819218 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:24.819276 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:24.819278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.819905 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.819906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:24.820821 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:24.820824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.821180 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.821181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:24.821256 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:24.821699 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:24.821705 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:24.823765 (+ 2060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 19:05:24.824015 9661 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:24.818853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:51577 (local address 127.0.196.254:34733)
0925 19:05:24.818914 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:24.818916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:24.818926 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:05:24.818988 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:24.818989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:24.819000 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:24.819046 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:24.819362 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.819823 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.820917 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.821103 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.821157 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:24.823935 (+ 2778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.193:51577: BlockingRecv error: recv got EOF from 127.0.196.193:51577 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 19:05:24 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:24 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:24.829205 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:24.824104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.193:60687)
0925 19:05:24.824264 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:24.824275 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:24.824353 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:24.824523 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:24.824524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:24.824664 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:24.824699 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:24.824700 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.825424 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.825425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:24.826227 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:24.826229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.826585 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.826586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:24.826672 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:24.827153 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:24.827159 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:24.829109 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":108,"threads_started":1}
W20260925 19:05:24.829252 9661 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:24.824186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:60687 (local address 127.0.196.254:34733)
0925 19:05:24.824271 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:24.824273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:24.824318 (+ 45us) server_negotiation.cc:407] Connection header received
0925 19:05:24.824429 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:24.824430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:24.824443 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:24.824501 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:24.824787 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.825343 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.826319 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.826501 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.826570 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:24.829188 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.193:60687: BlockingRecv error: recv got EOF from 127.0.196.193:60687 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 19:05:24.829381 9658 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 19:05:24.898341 9662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:24.898625 9662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:24.898700 9662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:24.899983 9662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:24.900064 9662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:24.900113 9662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:05:24.901674 9662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/oryx_127.0.196.194.keytab
--principal=oryx/127.0.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:36427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=42841
--tserver_master_addrs=127.0.196.254:34733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:24.901979 9662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:24.902230 9662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:24.907634 9668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:24.907689 9669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:24.907646 9671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:24.907905 9662 server_base.cc:1034] running on GCE node
I20260925 19:05:24.908152 9662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:24.908421 9662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:24.909571 9662 hybrid_clock.cc:648] HybridClock initialized: now 1790363124909556 us; error 29 us; skew 500 ppm
Sep 25 19:05:24 dist-test-slave-rn98 krb5kdc[8927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363124, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:24.912480 9662 init.cc:377] Logged in from keytab as oryx/127.0.196.194@KRBTEST.COM (short username oryx)
I20260925 19:05:24.912988 9662 webserver.cc:460] Webserver started at http://127.0.196.194:42841/ using document root <none> and password file <none>
I20260925 19:05:24.913255 9662 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:24.913313 9662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:24.914757 9662 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:24.915381 9678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:24.915563 9662 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:05:24.915654 9662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "a4ca9804bd4549a38ad9c4239d76a1d5"
format_stamp: "Formatted at 2026-09-25 19:05:24 on dist-test-slave-rn98"
server_key: "b542992d18a03b8edf80c8d268f5fb6a"
server_key_iv: "db1704adfd3ff0550c8af8f0b2e1be09"
server_key_version: "encryptionkey@0"
I20260925 19:05:24.916024 9662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:24.929344 9662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:24.931655 9662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:24.931841 9662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:24.932123 9662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:24.932471 9662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:24.932504 9662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:24.932528 9662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:24.932577 9662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:24.938623 9662 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:36427
I20260925 19:05:24.938699 9791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:36427 every 8 connection(s)
I20260925 19:05:24.939049 9662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:05:24 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:24 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:24.946259 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 9662
I20260925 19:05:24.946564 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:34511
--local_ip_for_outbound_sockets=127.0.196.195
--tserver_master_addrs=127.0.196.254:34733
--webserver_port=46853
--webserver_interface=127.0.196.195
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.196.195
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/oryx_127.0.196.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:24.946982 9661 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:24.939892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:49767 (local address 127.0.196.254:34733)
0925 19:05:24.940118 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:24.940121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:24.940293 (+ 172us) server_negotiation.cc:407] Connection header received
0925 19:05:24.940846 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:24.940847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:24.940863 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:24.940929 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:24.941995 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.942496 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.943588 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.943808 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.943917 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:24.946925 (+ 3008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.194:49767: BlockingRecv error: recv got EOF from 127.0.196.194:49767 (error 108)
Metrics: {"server-negotiator.queue_time_us":194}
I20260925 19:05:24.947049 9794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:24.939807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.194:49767)
0925 19:05:24.940151 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:24.940205 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:24.940731 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:24.941017 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:24.941026 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:24.941625 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:24.941828 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:24.941833 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.942653 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.942654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:24.943486 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:24.943488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.943925 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.943927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:24.944065 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:24.944585 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:24.944594 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:24.946834 (+ 2240us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":94,"threads_started":1}
W20260925 19:05:24.947293 9792 heartbeater.cc:644] Failed to heartbeat to 127.0.196.254:34733 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.196.254:34733: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Sep 25 19:05:24 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:24 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:24.951815 9794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:24.947548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.194:51223)
0925 19:05:24.947583 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:24.947593 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:24.947634 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:24.947746 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:24.947746 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:24.947886 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:24.947933 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:24.947935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.948567 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.948568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:24.949082 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:24.949083 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.949446 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.949447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:24.949519 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:24.950006 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:24.950010 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:24.951729 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":15}
W20260925 19:05:24.951866 9661 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:24.947600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:51223 (local address 127.0.196.254:34733)
0925 19:05:24.947651 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:24.947653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:24.947661 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:05:24.947687 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:24.947688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:24.947696 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:24.947745 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:24.948017 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.948487 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.949167 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.949364 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.949439 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:24.951803 (+ 2364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.194:51223: BlockingRecv error: recv got EOF from 127.0.196.194:51223 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 19:05:24 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:24 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:24.956600 9794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:24.952109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.194:38429)
0925 19:05:24.952148 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:24.952158 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:24.952197 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:24.952421 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:24.952422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:24.952543 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:24.952568 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:24.952569 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.953190 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.953191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:24.953693 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:24.953694 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.954058 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.954059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:24.954110 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:24.954708 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:24.954713 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:24.956523 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":22}
W20260925 19:05:24.956646 9661 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:24.952261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:38429 (local address 127.0.196.254:34733)
0925 19:05:24.952321 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:24.952323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:24.952329 (+ 6us) server_negotiation.cc:407] Connection header received
0925 19:05:24.952345 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:24.952346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:24.952353 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:24.952418 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:24.952641 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.953110 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.953776 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:24.953985 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:24.954061 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:24.956598 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.194:38429: BlockingRecv error: recv got EOF from 127.0.196.194:38429 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 19:05:24.957187 9792 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 19:05:25.031008 9795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:25.031263 9795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:25.031306 9795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:25.032569 9795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:25.032651 9795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:25.032701 9795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:05:25.034272 9795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/krb5kdc/oryx_127.0.196.195.keytab
--principal=oryx/127.0.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:34511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=46853
--tserver_master_addrs=127.0.196.254:34733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:25.034575 9795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:25.034843 9795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:25.040372 9800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:25.040344 9803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:25.040347 9801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:25.040568 9795 server_base.cc:1034] running on GCE node
I20260925 19:05:25.040781 9795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:25.041061 9795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:25.042244 9795 hybrid_clock.cc:648] HybridClock initialized: now 1790363125042235 us; error 47 us; skew 500 ppm
Sep 25 19:05:25 dist-test-slave-rn98 krb5kdc[8927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363125, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:25.045116 9795 init.cc:377] Logged in from keytab as oryx/127.0.196.195@KRBTEST.COM (short username oryx)
I20260925 19:05:25.045629 9795 webserver.cc:460] Webserver started at http://127.0.196.195:46853/ using document root <none> and password file <none>
I20260925 19:05:25.045877 9795 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:25.045965 9795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:25.047421 9795 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:05:25.048130 9810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:25.048341 9795 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:25.048444 9795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "4bd8b3cadbbb46f3ad02b5c695bcaae6"
format_stamp: "Formatted at 2026-09-25 19:05:24 on dist-test-slave-rn98"
server_key: "963be4367eb6ac2507b17c17f632241c"
server_key_iv: "594530f73e9dec123f6dea2ecc7cd7f1"
server_key_version: "encryptionkey@0"
I20260925 19:05:25.048810 9795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:25.065245 9795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:25.067404 9795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:25.067584 9795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:25.067828 9795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:25.068140 9795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:25.068197 9795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:25.068243 9795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:25.068277 9795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:25.074780 9795 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:34511
I20260925 19:05:25.074824 9923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:34511 every 8 connection(s)
I20260925 19:05:25.075203 9795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
Sep 25 19:05:25 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:25 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:25.082093 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 9795
W20260925 19:05:25.083058 9661 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:25.076096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:53703 (local address 127.0.196.254:34733)
0925 19:05:25.076170 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:25.076172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:25.076358 (+ 186us) server_negotiation.cc:407] Connection header received
0925 19:05:25.076915 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:25.076916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:25.076928 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:25.076993 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:25.077960 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:25.078456 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:25.079538 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:25.079732 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:25.079804 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:25.082979 (+ 3175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.195:53703: BlockingRecv error: recv got EOF from 127.0.196.195:53703 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 19:05:25.083148 9926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:25.076039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.195:53703)
0925 19:05:25.076217 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:25.076261 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:25.076817 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:25.077052 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:25.077055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:25.077635 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:25.077839 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:25.077844 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:25.078560 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:25.078561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:25.079415 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:25.079417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:25.079816 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:25.079818 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:25.079933 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:25.080459 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:25.080468 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:25.082893 (+ 2425us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
W20260925 19:05:25.083364 9924 heartbeater.cc:644] Failed to heartbeat to 127.0.196.254:34733 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.196.254:34733: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Sep 25 19:05:25 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:25 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:25.089290 9926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:25.083598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.195:55791)
0925 19:05:25.083642 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:25.083652 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:25.083689 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:25.083834 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:25.083835 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:25.083979 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:25.084031 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:25.084032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:25.084856 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:25.084857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:25.085682 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:25.085684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:25.086210 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:25.086212 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:25.086307 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:25.086950 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:25.086954 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:25.089192 (+ 2238us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 19:05:25.089324 9661 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:25.083647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:55791 (local address 127.0.196.254:34733)
0925 19:05:25.083679 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:25.083681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:25.083689 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:05:25.083755 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:25.083756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:25.083763 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:25.083809 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:25.084090 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:25.084770 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:25.085764 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:25.086077 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:25.086153 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:25.089267 (+ 3114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.195:55791: BlockingRecv error: recv got EOF from 127.0.196.195:55791 (error 108)
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":15}
I20260925 19:05:25.091547 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:25.082709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49626 (local address 127.0.196.254:34733)
0925 19:05:25.082806 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:25.082808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:25.082855 (+ 47us) server_negotiation.cc:407] Connection header received
0925 19:05:25.082944 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:25.082945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:25.082958 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:25.083018 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:25.083973 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:25.084770 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:25.085836 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:25.086109 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:25.086168 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:25.087999 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:25.088005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:25.088006 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:25.088021 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:25.090082 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:25.090466 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:25.090470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:25.090471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:25.090582 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:25.090816 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:25.090817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:25.090817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:25.091060 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:25.091208 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:25.091332 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 19:05:25.091442 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":53,"threads_started":1}
Sep 25 19:05:25 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:25 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:25.093967 9926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:25.089674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.195:38365)
0925 19:05:25.089731 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:25.089743 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:25.089789 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:25.089906 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:25.089907 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:25.090069 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:25.090101 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:25.090103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:25.090826 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:25.090826 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:25.091325 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:25.091326 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:25.091780 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:25.091781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:25.091846 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:25.092274 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:25.092279 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:25.093918 (+ 1639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 19:05:25.094038 9661 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:25.089753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:38365 (local address 127.0.196.254:34733)
0925 19:05:25.089791 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:25.089793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:25.089801 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:05:25.089846 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:25.089847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:25.089856 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:25.089904 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:25.090185 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:25.090741 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:25.091402 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:25.091709 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:25.091771 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:25.093994 (+ 2223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.195:38365: BlockingRecv error: recv got EOF from 127.0.196.195:38365 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 19:05:25.094209 9924 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 19:05:25 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:25 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:25.834988 9929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:25.829863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.193:48233)
0925 19:05:25.830025 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:25.830044 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:25.830100 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:25.830265 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:25.830266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:25.830440 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:25.830487 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:25.830489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:25.831233 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:25.831234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:25.831929 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:25.831932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:25.832301 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:25.832302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:25.832354 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:25.832813 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:25.832818 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:25.834920 (+ 2102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":113,"threads_started":1}
W20260925 19:05:25.835165 9930 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:25.829962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:48233 (local address 127.0.196.254:34733)
0925 19:05:25.830108 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:25.830109 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:25.830121 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:05:25.830145 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:25.830145 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:25.830155 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:25.830227 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:25.830557 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:25.831115 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:25.832023 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:25.832217 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:25.832276 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:25.835063 (+ 2787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.193:48233: BlockingRecv error: recv got EOF from 127.0.196.193:48233 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":91,"threads_started":1}
Sep 25 19:05:25 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:25 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:25.962564 9931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:25.957634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.194:51971)
0925 19:05:25.957787 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:25.957804 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:25.957861 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:25.958028 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:25.958028 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:25.958192 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:25.958239 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:25.958240 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:25.959054 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:25.959055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:25.959606 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:25.959609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:25.960066 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:25.960067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:25.960107 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:25.960558 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:25.960563 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:25.962496 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
W20260925 19:05:25.962754 9930 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:25.957705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:51971 (local address 127.0.196.254:34733)
0925 19:05:25.957771 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:25.957773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:25.957861 (+ 88us) server_negotiation.cc:407] Connection header received
0925 19:05:25.957936 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:25.957937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:25.957948 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:25.958005 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:25.958300 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:25.958949 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:25.959705 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:25.959962 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:25.960029 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:25.962670 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.194:51971: BlockingRecv error: recv got EOF from 127.0.196.194:51971 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 19:05:26 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:26 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:26.099725 9932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:26.094751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.195:49997)
0925 19:05:26.094877 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:26.094895 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:26.095027 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:26.095176 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:26.095176 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:26.095339 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:26.095388 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:26.095389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:26.096142 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:26.096142 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:26.096681 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:26.096682 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:26.097084 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:26.097085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:26.097125 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:26.097585 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:26.097589 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:26.099659 (+ 2070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
W20260925 19:05:26.099897 9930 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:26.094833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:49997 (local address 127.0.196.254:34733)
0925 19:05:26.094908 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:26.094911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:26.095026 (+ 115us) server_negotiation.cc:407] Connection header received
0925 19:05:26.095075 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:26.095075 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:26.095088 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:26.095157 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:26.095455 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:26.096060 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:26.096756 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:26.097013 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:26.097074 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:26.099755 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.195:49997: BlockingRecv error: recv got EOF from 127.0.196.195:49997 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 19:05:26 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:26 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:26.840337 9933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:26.835547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.193:42261)
0925 19:05:26.835681 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:26.835699 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:26.835752 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:26.836030 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:26.836031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:26.836191 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:26.836240 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:26.836241 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:26.836972 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:26.836972 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:26.837510 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:26.837511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:26.837896 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:26.837897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:26.837946 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:26.838471 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:26.838476 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:26.840242 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
W20260925 19:05:26.840556 9934 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:26.835663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:42261 (local address 127.0.196.254:34733)
0925 19:05:26.835875 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:26.835877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:26.835886 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:05:26.835911 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:26.835912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:26.835923 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:26.836025 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:26.836336 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:26.836894 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:26.837587 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:26.837778 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:26.837887 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:26.840392 (+ 2505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.193:42261: BlockingRecv error: recv got EOF from 127.0.196.193:42261 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":89,"threads_started":1}
Sep 25 19:05:26 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:26 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:26.968660 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:26.963161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.194:56255)
0925 19:05:26.963329 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:26.963341 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:26.963434 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:26.963615 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:26.963615 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:26.963904 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:26.963956 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:26.963959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:26.964725 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:26.964726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:26.965321 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:26.965323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:26.965734 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:26.965735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:26.965811 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:26.966352 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:26.966358 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:26.968551 (+ 2193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":91,"threads_started":1}
W20260925 19:05:26.968724 9934 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:26.963204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:56255 (local address 127.0.196.254:34733)
0925 19:05:26.963272 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:26.963274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:26.963414 (+ 140us) server_negotiation.cc:407] Connection header received
0925 19:05:26.963508 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:26.963509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:26.963532 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:26.963607 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:26.964061 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:26.964626 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:26.965417 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:26.965637 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:26.965735 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:26.968655 (+ 2920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.194:56255: BlockingRecv error: recv got EOF from 127.0.196.194:56255 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 19:05:27 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:27 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:27.105474 9936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:27.100310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.195:47163)
0925 19:05:27.100449 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:27.100462 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:27.100535 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:27.100725 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:27.100727 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:27.100952 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:27.101014 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:27.101016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:27.102006 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:27.102007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:27.102544 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:27.102546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:27.102948 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:27.102949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:27.103006 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:27.103547 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:27.103553 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:27.105375 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":105,"threads_started":1}
W20260925 19:05:27.105530 9934 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:27.100493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:47163 (local address 127.0.196.254:34733)
0925 19:05:27.100538 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:27.100540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:27.100586 (+ 46us) server_negotiation.cc:407] Connection header received
0925 19:05:27.100613 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:27.100614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:27.100630 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:27.100707 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:27.101114 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:27.101879 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:27.102664 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:27.102869 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:27.102950 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:27.105465 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.195:47163: BlockingRecv error: recv got EOF from 127.0.196.195:47163 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 25 19:05:27 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:27 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:27.846288 9937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:27.841059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.193:49839)
0925 19:05:27.841192 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:27.841202 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:27.841274 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:27.841439 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:27.841440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:27.841619 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:27.841667 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:27.841669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:27.842495 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:27.842496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:27.843230 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:27.843234 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:27.843685 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:27.843686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:27.843736 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:27.844375 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:27.844381 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:27.846214 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
W20260925 19:05:27.846503 9938 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:27.841127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:49839 (local address 127.0.196.254:34733)
0925 19:05:27.841216 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:27.841218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:27.841255 (+ 37us) server_negotiation.cc:407] Connection header received
0925 19:05:27.841320 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:27.841321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:27.841331 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:27.841389 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:27.841753 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:27.842389 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:27.843359 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:27.843599 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:27.843667 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:27.846420 (+ 2753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.193:49839: BlockingRecv error: recv got EOF from 127.0.196.193:49839 (error 108)
Metrics: {"server-negotiator.queue_time_us":67,"thread_start_us":43,"threads_started":1}
Sep 25 19:05:27 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:27 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:27.974736 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:27.969327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.194:46683)
0925 19:05:27.969500 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:27.969513 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:27.969586 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:27.969775 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:27.969775 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:27.969948 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:27.969999 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:27.970000 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:27.970780 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:27.970781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:27.971363 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:27.971365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:27.971763 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:27.971764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:27.971803 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:27.972367 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:27.972372 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:27.974654 (+ 2282us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":108,"threads_started":1}
W20260925 19:05:27.974980 9938 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:27.969458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:46683 (local address 127.0.196.254:34733)
0925 19:05:27.969511 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:27.969514 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:27.969574 (+ 60us) server_negotiation.cc:407] Connection header received
0925 19:05:27.969650 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:27.969651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:27.969666 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:27.969748 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:27.970088 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:27.970679 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:27.971462 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:27.971687 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:27.971753 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:27.974918 (+ 3165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.194:46683: BlockingRecv error: recv got EOF from 127.0.196.194:46683 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 25 19:05:28 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:28 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:28.112192 9940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:28.106133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.195:33005)
0925 19:05:28.106333 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:28.106346 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:28.106418 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:28.106598 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:28.106599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:28.106853 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:28.106920 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:28.106923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:28.107747 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:28.107749 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:28.108651 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:28.108653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:28.109161 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:28.109161 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:28.109200 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:28.109705 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:28.109718 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:28.112081 (+ 2363us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":73,"threads_started":1}
W20260925 19:05:28.112231 9938 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:28.106194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:33005 (local address 127.0.196.254:34733)
0925 19:05:28.106271 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:28.106273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:28.106397 (+ 124us) server_negotiation.cc:407] Connection header received
0925 19:05:28.106513 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:28.106514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:28.106528 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:28.106581 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:28.107011 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:28.107656 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:28.108736 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:28.109077 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:28.109161 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:28.112164 (+ 3003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.195:33005: BlockingRecv error: recv got EOF from 127.0.196.195:33005 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 19:05:28 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:28 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:28.855016 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:28.846814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.193:53987)
0925 19:05:28.846970 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:28.846984 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:28.847071 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:28.847222 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:28.847223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:28.847402 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:28.847449 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:28.847450 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:28.848160 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:28.848161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:28.848695 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:28.848697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:28.849142 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:28.849144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:28.849209 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:28.853074 (+ 3865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:28.853079 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:28.854947 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":54,"threads_started":1}
W20260925 19:05:28.855156 9942 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:28.846928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:53987 (local address 127.0.196.254:34733)
0925 19:05:28.847070 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:28.847071 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:28.847083 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:05:28.847127 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:28.847128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:28.847139 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:28.847200 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:28.847546 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:28.848071 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:28.848825 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:28.849036 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:28.849102 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:28.855064 (+ 5962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.193:53987: BlockingRecv error: recv got EOF from 127.0.196.193:53987 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":70,"threads_started":1}
Sep 25 19:05:28 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:28 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:28.980132 9943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:28.975277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.194:42283)
0925 19:05:28.975447 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:28.975459 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:28.975554 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:28.975696 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:28.975697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:28.975891 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:28.975963 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:28.975965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:28.976763 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:28.976764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:28.977297 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:28.977298 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:28.977677 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:28.977678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:28.977738 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:28.978239 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:28.978244 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:28.980033 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":88,"threads_started":1}
W20260925 19:05:28.980181 9942 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:28.975350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:42283 (local address 127.0.196.254:34733)
0925 19:05:28.975416 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:28.975419 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:28.975533 (+ 114us) server_negotiation.cc:407] Connection header received
0925 19:05:28.975624 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:28.975626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:28.975638 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:28.975693 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:28.976053 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:28.976668 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:28.977388 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:28.977587 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:28.977674 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:28.980115 (+ 2441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.194:42283: BlockingRecv error: recv got EOF from 127.0.196.194:42283 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 25 19:05:29 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:29 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:29.118227 9944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:29.112798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.195:54975)
0925 19:05:29.112951 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:29.112964 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:29.113046 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:29.113201 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:29.113202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:29.113360 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:29.113411 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:29.113412 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:29.114585 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:29.114586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:29.115322 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:29.115325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:29.115898 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:29.115900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:29.115948 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:29.116396 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:29.116401 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:29.118145 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
W20260925 19:05:29.118254 9942 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:29.112906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:54975 (local address 127.0.196.254:34733)
0925 19:05:29.112967 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:29.112975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:29.113054 (+ 79us) server_negotiation.cc:407] Connection header received
0925 19:05:29.113107 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:29.113109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:29.113125 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:29.113188 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:29.113569 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:29.114483 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:29.115419 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:29.115793 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:29.115879 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:29.118204 (+ 2325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.195:54975: BlockingRecv error: recv got EOF from 127.0.196.195:54975 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 25 19:05:29 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:29 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:29.860632 9945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:29.855577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.193:36705)
0925 19:05:29.855722 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:29.855747 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:29.855878 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:29.856050 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:29.856051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:29.856239 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:29.856290 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:29.856292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:29.856950 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:29.856952 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:29.857538 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:29.857539 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:29.857942 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:29.857943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:29.857986 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:29.858586 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:29.858591 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:29.860571 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
W20260925 19:05:29.860769 9946 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:29.855656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:36705 (local address 127.0.196.254:34733)
0925 19:05:29.855776 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:29.855778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:29.855857 (+ 79us) server_negotiation.cc:407] Connection header received
0925 19:05:29.855958 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:29.855964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:29.855979 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:29.856041 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:29.856370 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:29.856867 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:29.857637 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:29.857863 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:29.857934 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:29.860676 (+ 2742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.193:36705: BlockingRecv error: recv got EOF from 127.0.196.193:36705 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":46,"threads_started":1}
Sep 25 19:05:29 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:29 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:29.985594 9947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:29.980706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.194:40349)
0925 19:05:29.980857 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:29.980870 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:29.980958 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:29.981099 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:29.981099 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:29.981269 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:29.981334 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:29.981336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:29.982021 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:29.982022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:29.982566 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:29.982567 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:29.983056 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:29.983057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:29.983107 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:29.983748 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:29.983753 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:29.985535 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":171,"threads_started":1}
W20260925 19:05:29.985742 9946 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:29.980841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:40349 (local address 127.0.196.254:34733)
0925 19:05:29.980893 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:29.980896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:29.980947 (+ 51us) server_negotiation.cc:407] Connection header received
0925 19:05:29.981012 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:29.981012 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:29.981022 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:29.981077 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:29.981418 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:29.981907 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:29.982667 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:29.982848 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:29.982898 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:29.985667 (+ 2769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.194:40349: BlockingRecv error: recv got EOF from 127.0.196.194:40349 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 25 19:05:30 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:30 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:30.124451 9948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:30.118805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.195:54027)
0925 19:05:30.118949 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:30.118960 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:30.119023 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:30.119211 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:30.119212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:30.119444 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:30.119491 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:30.119494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:30.120184 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:30.120185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:30.121053 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:30.121056 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:30.121441 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:30.121442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:30.121497 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:30.122127 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:30.122132 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:30.124326 (+ 2194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
W20260925 19:05:30.124468 9946 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:30.118891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:54027 (local address 127.0.196.254:34733)
0925 19:05:30.119017 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:30.119019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:30.119091 (+ 72us) server_negotiation.cc:407] Connection header received
0925 19:05:30.119120 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:30.119121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:30.119134 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:30.119188 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:30.119573 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:30.120109 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:30.121135 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:30.121356 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:30.121443 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:30.124419 (+ 2976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.195:54027: BlockingRecv error: recv got EOF from 127.0.196.195:54027 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 25 19:05:30 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:30 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:30.866062 9949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:30.861215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.193:55215)
0925 19:05:30.861383 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:30.861396 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:30.861465 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:30.861636 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:30.861637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:30.861818 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:30.861875 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:30.861877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:30.862572 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:30.862573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:30.863130 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:30.863133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:30.863519 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:30.863519 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:30.863596 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:30.864076 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:30.864081 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:30.865963 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":64,"threads_started":1}
W20260925 19:05:30.866111 9950 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:30.861264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:55215 (local address 127.0.196.254:34733)
0925 19:05:30.861373 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:30.861375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:30.861458 (+ 83us) server_negotiation.cc:407] Connection header received
0925 19:05:30.861522 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:30.861523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:30.861537 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:30.861617 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:30.861965 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:30.862479 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:30.863225 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:30.863425 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:30.863518 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:30.866040 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.193:55215: BlockingRecv error: recv got EOF from 127.0.196.193:55215 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":48,"threads_started":1}
Sep 25 19:05:30 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:30 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:30.990756 9951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:30.986069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.194:57109)
0925 19:05:30.986205 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:30.986216 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:30.986277 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:30.986393 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:30.986394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:30.986550 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:30.986599 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:30.986600 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:30.987350 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:30.987351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:30.987880 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:30.987882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:30.988211 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:30.988211 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:30.988249 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:30.988760 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:30.988765 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:30.990682 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
W20260925 19:05:30.990888 9950 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:30.986116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:57109 (local address 127.0.196.254:34733)
0925 19:05:30.986167 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:30.986170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:30.986285 (+ 115us) server_negotiation.cc:407] Connection header received
0925 19:05:30.986313 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:30.986314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:30.986325 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:30.986375 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:30.986726 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:30.987269 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:30.987960 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:30.988146 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:30.988202 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:30.990797 (+ 2595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.194:57109: BlockingRecv error: recv got EOF from 127.0.196.194:57109 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 25 19:05:31 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:31 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:31.129390 9952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:31.124913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.195:42905)
0925 19:05:31.125068 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:31.125087 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:31.125138 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:31.125291 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:31.125291 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:31.125459 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:31.125507 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:31.125509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:31.126182 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:31.126183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:31.126779 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:31.126781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:31.127100 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:31.127100 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:31.127139 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:31.127613 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:31.127618 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:31.129330 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":104,"threads_started":1}
W20260925 19:05:31.129591 9950 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:31.125010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:42905 (local address 127.0.196.254:34733)
0925 19:05:31.125111 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:31.125113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:31.125127 (+ 14us) server_negotiation.cc:407] Connection header received
0925 19:05:31.125194 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:31.125195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:31.125207 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:31.125271 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:31.125589 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:31.126113 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:31.126844 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:31.127035 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:31.127089 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:31.129494 (+ 2405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.195:42905: BlockingRecv error: recv got EOF from 127.0.196.195:42905 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 25 19:05:31 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:31 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:31.870973 9953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:31.866565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.193:58167)
0925 19:05:31.866724 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:31.866736 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:31.866796 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:31.866938 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:31.866939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:31.867100 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:31.867150 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:31.867152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:31.867808 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:31.867810 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:31.868348 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:31.868349 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:31.868675 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:31.868675 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:31.868715 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:31.869189 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:31.869193 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:31.870913 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":100,"threads_started":1}
W20260925 19:05:31.871114 9954 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:31.866675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:58167 (local address 127.0.196.254:34733)
0925 19:05:31.866803 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:31.866805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:31.866816 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:05:31.866839 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:31.866839 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:31.866850 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:31.866930 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:31.867215 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:31.867732 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:31.868425 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:31.868612 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:31.868663 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:31.871004 (+ 2341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.193:58167: BlockingRecv error: recv got EOF from 127.0.196.193:58167 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
Sep 25 19:05:31 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:31 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:31.996212 9955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:31.991357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.194:60391)
0925 19:05:31.991540 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:31.991558 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:31.991617 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:31.991760 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:31.991761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:31.991943 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:31.991993 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:31.991995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:31.992679 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:31.992680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:31.993201 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:31.993203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:31.993544 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:31.993544 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:31.993588 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:31.994194 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:31.994199 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:31.996154 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":125,"threads_started":1}
W20260925 19:05:31.996342 9954 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:31.991442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:60391 (local address 127.0.196.254:34733)
0925 19:05:31.991518 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:31.991520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:31.991620 (+ 100us) server_negotiation.cc:407] Connection header received
0925 19:05:31.991680 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:31.991681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:31.991692 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:31.991736 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:31.992101 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:31.992606 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:31.993280 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:31.993470 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:31.993526 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:31.996260 (+ 2734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.194:60391: BlockingRecv error: recv got EOF from 127.0.196.194:60391 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 19:05:32 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:32 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:32.134519 9956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:32.129908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.195:59523)
0925 19:05:32.130064 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:32.130073 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:32.130114 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:32.130279 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:32.130279 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:32.130444 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:32.130491 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:32.130492 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:32.131241 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:32.131241 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:32.131764 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:32.131766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:32.132154 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:32.132154 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:32.132193 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:32.132673 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:32.132677 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:32.134458 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":70,"threads_started":1}
W20260925 19:05:32.134624 9954 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:32.130066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:59523 (local address 127.0.196.254:34733)
0925 19:05:32.130136 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:32.130138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:32.130147 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:05:32.130172 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:32.130173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:32.130185 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:32.130252 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:32.130563 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:32.131161 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:32.131878 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:32.132081 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:32.132145 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:32.134554 (+ 2409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.195:59523: BlockingRecv error: recv got EOF from 127.0.196.195:59523 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 19:05:32 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:32 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:32.876708 9957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:32.871487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.193:36019)
0925 19:05:32.871639 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:32.871658 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:32.871712 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:32.871992 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:32.871993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:32.872226 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:32.872283 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:32.872285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:32.873009 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:32.873010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:32.873630 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:32.873633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:32.874143 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:32.874145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:32.874226 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:32.874720 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:32.874725 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:32.876648 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":96,"threads_started":1}
W20260925 19:05:32.876842 9958 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:32.871643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:36019 (local address 127.0.196.254:34733)
0925 19:05:32.871795 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:32.871797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:32.871806 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:05:32.871831 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:32.871832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:32.871842 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:32.871921 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:32.872374 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:32.872917 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:32.873734 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:32.874042 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:32.874135 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:32.876751 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.193:36019: BlockingRecv error: recv got EOF from 127.0.196.193:36019 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
Sep 25 19:05:32 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:32 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:33.001710 9959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:32.996735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.194:50309)
0925 19:05:32.996943 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:32.996955 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:32.997041 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:32.997201 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:32.997202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:32.997386 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:32.997430 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:32.997432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:32.998252 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:32.998253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:32.998822 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:32.998824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:32.999199 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:32.999200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:32.999256 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:32.999773 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:32.999778 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:33.001649 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":90,"threads_started":1}
W20260925 19:05:33.001863 9958 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:32.996823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:50309 (local address 127.0.196.254:34733)
0925 19:05:32.996894 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:32.996896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:32.997023 (+ 127us) server_negotiation.cc:407] Connection header received
0925 19:05:32.997099 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:32.997100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:32.997113 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:32.997181 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:32.997511 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:32.998155 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:32.998920 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:32.999117 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:32.999188 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:33.001774 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.194:50309: BlockingRecv error: recv got EOF from 127.0.196.194:50309 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 19:05:33 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:33 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:33.140514 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:33.135099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.195:42765)
0925 19:05:33.135310 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:33.135326 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:33.135449 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:33.135627 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:33.135629 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:33.135852 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:33.135902 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:33.135904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:33.136672 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:33.136674 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:33.137264 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:33.137266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:33.137666 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:33.137667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:33.137728 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:33.138250 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:33.138256 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:33.140416 (+ 2160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":80,"threads_started":1}
W20260925 19:05:33.140657 9958 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:33.135225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:42765 (local address 127.0.196.254:34733)
0925 19:05:33.135332 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:33.135335 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:33.135425 (+ 90us) server_negotiation.cc:407] Connection header received
0925 19:05:33.135539 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:33.135540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:33.135555 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:33.135609 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:33.136010 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:33.136568 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:33.137375 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:33.137586 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:33.137668 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:33.140582 (+ 2914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.195:42765: BlockingRecv error: recv got EOF from 127.0.196.195:42765 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 19:05:33 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:33 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:33.886380 9961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:33.877198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.193:48557)
0925 19:05:33.877340 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:33.877368 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:33.877424 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:33.877563 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:33.877564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:33.877727 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:33.877774 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:33.877775 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:33.878433 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:33.878435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:33.879190 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:33.879199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:33.879623 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:33.879624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:33.879682 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:33.884305 (+ 4623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:33.884312 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:33.886274 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
W20260925 19:05:33.886445 9962 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:33.877282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:48557 (local address 127.0.196.254:34733)
0925 19:05:33.877420 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:33.877422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:33.877432 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:05:33.877464 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:33.877465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:33.877476 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:33.877547 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:33.877840 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:33.878340 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:33.879310 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:33.879518 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:33.879623 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:33.886371 (+ 6748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.193:48557: BlockingRecv error: recv got EOF from 127.0.196.193:48557 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
Sep 25 19:05:34 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:34 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:34.007238 9963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:34.002264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.194:54131)
0925 19:05:34.002436 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:34.002448 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:34.002521 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:34.002704 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:34.002705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:34.002939 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:34.002986 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:34.002988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:34.003686 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:34.003687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:34.004216 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:34.004218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:34.004595 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:34.004596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:34.004657 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:34.005202 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:34.005207 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:34.007126 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":105,"threads_started":1}
W20260925 19:05:34.007294 9962 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:34.002478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:54131 (local address 127.0.196.254:34733)
0925 19:05:34.002552 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:34.002554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:34.002563 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:05:34.002590 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:34.002591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:34.002603 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:34.002696 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:34.003081 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:34.003596 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:34.004310 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:34.004508 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:34.004595 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:34.007216 (+ 2621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.194:54131: BlockingRecv error: recv got EOF from 127.0.196.194:54131 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 19:05:34 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:34 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:34.146785 9964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:34.141217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.195:57275)
0925 19:05:34.141404 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:34.141416 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:34.141490 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:34.141667 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:34.141668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:34.141832 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:34.141880 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:34.141882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:34.142734 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:34.142735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:34.143608 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:34.143611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:34.144034 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:34.144035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:34.144098 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:34.144649 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:34.144655 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:34.146658 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":121,"threads_started":1}
W20260925 19:05:34.146824 9962 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:34.141348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:57275 (local address 127.0.196.254:34733)
0925 19:05:34.141428 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:34.141430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:34.141477 (+ 47us) server_negotiation.cc:407] Connection header received
0925 19:05:34.141557 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:34.141558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:34.141570 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:34.141617 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:34.142019 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:34.142577 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:34.143706 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:34.143949 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:34.144036 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:34.146747 (+ 2711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.195:57275: BlockingRecv error: recv got EOF from 127.0.196.195:57275 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 19:05:34 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:34 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:34.892182 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:34.887034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.193:33355)
0925 19:05:34.887176 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:34.887187 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:34.887246 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:34.887437 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:34.887438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:34.887608 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:34.887660 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:34.887662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:34.888584 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:34.888586 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:34.889150 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:34.889152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:34.889508 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:34.889509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:34.889566 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:34.890098 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:34.890103 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:34.892109 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
W20260925 19:05:34.892355 9966 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:34.887143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:33355 (local address 127.0.196.254:34733)
0925 19:05:34.887299 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:34.887301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:34.887310 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:05:34.887333 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:34.887334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:34.887344 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:34.887420 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:34.887912 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:34.888477 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:34.889237 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:34.889435 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:34.889492 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:34.892233 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.193:33355: BlockingRecv error: recv got EOF from 127.0.196.193:33355 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
Sep 25 19:05:35 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:35 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:35.012969 9967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:35.007746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.194:42887)
0925 19:05:35.007965 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:35.007978 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:35.008087 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:35.008270 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:35.008271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:35.008470 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:35.008518 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:35.008520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:35.009289 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:35.009290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:35.009900 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:35.009902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:35.010324 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:35.010324 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:35.010388 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:35.010873 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:35.010879 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:35.012859 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":100,"threads_started":1}
W20260925 19:05:35.012987 9966 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:35.007837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:42887 (local address 127.0.196.254:34733)
0925 19:05:35.007955 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:35.007958 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:35.008066 (+ 108us) server_negotiation.cc:407] Connection header received
0925 19:05:35.008158 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:35.008159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:35.008172 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:35.008258 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:35.008605 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:35.009179 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:35.009995 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:35.010230 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:35.010321 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:35.012921 (+ 2600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.194:42887: BlockingRecv error: recv got EOF from 127.0.196.194:42887 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 25 19:05:35 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:35 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:35.152724 9968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:35.147368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.195:51481)
0925 19:05:35.147517 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:35.147531 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:35.147599 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:35.147768 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:35.147769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:35.147969 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:35.148075 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:35.148077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:35.149186 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:35.149187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:35.149706 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:35.149707 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:35.150071 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:35.150071 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:35.150109 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:35.150602 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:35.150608 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:35.152630 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":60,"threads_started":1}
W20260925 19:05:35.152787 9966 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:35.147556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:51481 (local address 127.0.196.254:34733)
0925 19:05:35.147616 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:35.147623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:35.147633 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:05:35.147656 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:35.147657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:35.147668 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:35.147719 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:35.148186 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:35.149105 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:35.149788 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:35.150007 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:35.150064 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:35.152724 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.195:51481: BlockingRecv error: recv got EOF from 127.0.196.195:51481 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 25 19:05:35 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:35 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:35.897768 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:35.892767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.193:39375)
0925 19:05:35.892902 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:35.892914 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:35.892977 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:35.893215 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:35.893216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:35.893392 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:35.893440 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:35.893441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:35.894123 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:35.894124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:35.894714 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:35.894716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:35.895106 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:35.895107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:35.895155 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:35.895635 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:35.895639 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:35.897612 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
W20260925 19:05:35.897970 9970 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:35.892910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:39375 (local address 127.0.196.254:34733)
0925 19:05:35.893064 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:35.893066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:35.893074 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:05:35.893100 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:35.893100 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:35.893110 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:35.893194 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:35.893535 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:35.894049 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:35.894818 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:35.895032 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:35.895094 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:35.897874 (+ 2780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.193:39375: BlockingRecv error: recv got EOF from 127.0.196.193:39375 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
Sep 25 19:05:36 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:36 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:36.018342 9971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:36.013529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.194:54797)
0925 19:05:36.013719 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:36.013732 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:36.013826 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:36.013981 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:36.013982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:36.014174 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:36.014221 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:36.014223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:36.014945 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:36.014946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:36.015488 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:36.015490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:36.015872 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:36.015873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:36.015930 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:36.016381 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:36.016385 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:36.018275 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":111,"threads_started":1}
W20260925 19:05:36.018395 9970 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:36.013633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:54797 (local address 127.0.196.254:34733)
0925 19:05:36.013710 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:36.013712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:36.013808 (+ 96us) server_negotiation.cc:407] Connection header received
0925 19:05:36.013898 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:36.013899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:36.013914 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:36.013977 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:36.014306 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:36.014867 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:36.015582 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:36.015780 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:36.015873 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:36.018348 (+ 2475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.194:54797: BlockingRecv error: recv got EOF from 127.0.196.194:54797 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 19:05:36 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:36 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:36.158118 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:36.153272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.195:56141)
0925 19:05:36.153456 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:36.153469 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:36.153543 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:36.153707 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:36.153708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:36.153940 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:36.153988 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:36.153989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:36.154768 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:36.154769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:36.155307 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:36.155308 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:36.155656 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:36.155657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:36.155730 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:36.156191 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:36.156195 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:36.158046 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":85,"threads_started":1}
W20260925 19:05:36.158215 9970 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:36.153362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:56141 (local address 127.0.196.254:34733)
0925 19:05:36.153442 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:36.153444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:36.153532 (+ 88us) server_negotiation.cc:407] Connection header received
0925 19:05:36.153609 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:36.153610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:36.153625 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:36.153697 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:36.154065 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:36.154688 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:36.155401 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:36.155590 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:36.155658 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:36.158146 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.195:56141: BlockingRecv error: recv got EOF from 127.0.196.195:56141 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 19:05:36 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:36 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:36.903064 9973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:36.898280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.193:57009)
0925 19:05:36.898444 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:36.898454 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:36.898502 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:36.898651 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:36.898652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:36.898881 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:36.898928 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:36.898930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:36.899606 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:36.899607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:36.900167 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:36.900169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:36.900556 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:36.900557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:36.900609 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:36.901170 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:36.901174 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:36.902999 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":81,"threads_started":1}
W20260925 19:05:36.903172 9974 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:36.898320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:57009 (local address 127.0.196.254:34733)
0925 19:05:36.898473 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:36.898475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:36.898487 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:05:36.898547 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:36.898547 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:36.898558 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:36.898644 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:36.899002 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:36.899523 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:36.900269 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:36.900469 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:36.900529 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:36.903093 (+ 2564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.193:57009: BlockingRecv error: recv got EOF from 127.0.196.193:57009 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":60,"threads_started":1}
Sep 25 19:05:37 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:37 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:37.023864 9975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:37.018974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.194:47051)
0925 19:05:37.019105 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:37.019118 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:37.019204 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:37.019388 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:37.019389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:37.019616 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:37.019686 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:37.019689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:37.020475 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:37.020476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:37.021043 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:37.021045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:37.021407 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:37.021408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:37.021449 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:37.021954 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:37.021959 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:37.023785 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
W20260925 19:05:37.024041 9974 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:37.019056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:47051 (local address 127.0.196.254:34733)
0925 19:05:37.019109 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:37.019112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:37.019232 (+ 120us) server_negotiation.cc:407] Connection header received
0925 19:05:37.019260 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:37.019261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:37.019273 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:37.019358 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:37.019755 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:37.020398 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:37.021130 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:37.021335 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:37.021398 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:37.023946 (+ 2548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.194:47051: BlockingRecv error: recv got EOF from 127.0.196.194:47051 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 25 19:05:37 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:37 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:37.163749 9976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:37.158707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.195:49051)
0925 19:05:37.158844 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:37.158861 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:37.158924 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:37.159083 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:37.159084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:37.159268 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:37.159315 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:37.159317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:37.160083 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:37.160084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:37.160667 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:37.160669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:37.161064 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:37.161065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:37.161108 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:37.161736 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:37.161742 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:37.163671 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
W20260925 19:05:37.163815 9974 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:37.158829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:49051 (local address 127.0.196.254:34733)
0925 19:05:37.158911 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:37.158913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:37.158933 (+ 20us) server_negotiation.cc:407] Connection header received
0925 19:05:37.158978 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:37.158979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:37.158997 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:37.159065 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:37.159406 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:37.159991 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:37.160803 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:37.160997 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:37.161056 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:37.163764 (+ 2708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.195:49051: BlockingRecv error: recv got EOF from 127.0.196.195:49051 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 19:05:37 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:37 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:37.908229 9977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:37.903593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.193:54497)
0925 19:05:37.903730 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:37.903741 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:37.903832 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:37.903958 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:37.903958 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:37.904127 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:37.904172 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:37.904173 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:37.904844 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:37.904845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:37.905397 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:37.905399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:37.905721 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:37.905722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:37.905765 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:37.906259 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:37.906264 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:37.908137 (+ 1873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
W20260925 19:05:37.908326 9978 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:37.903684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:54497 (local address 127.0.196.254:34733)
0925 19:05:37.903805 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:37.903807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:37.903819 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:05:37.903881 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:37.903882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:37.903893 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:37.903952 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:37.904236 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:37.904730 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:37.905470 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:37.905660 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:37.905710 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:37.908254 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.193:54497: BlockingRecv error: recv got EOF from 127.0.196.193:54497 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":72,"threads_started":1}
Sep 25 19:05:38 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:38 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:38.029002 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:38.024376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.194:52175)
0925 19:05:38.024576 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:38.024587 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:38.024643 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:38.024801 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:38.024802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:38.024977 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:38.025025 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:38.025027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:38.025690 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:38.025691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:38.026234 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:38.026236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:38.026582 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:38.026583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:38.026680 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:38.027132 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:38.027136 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:38.028943 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":95,"threads_started":1}
W20260925 19:05:38.029153 9978 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:38.024512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:52175 (local address 127.0.196.254:34733)
0925 19:05:38.024555 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:38.024557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:38.024628 (+ 71us) server_negotiation.cc:407] Connection header received
0925 19:05:38.024698 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:38.024699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:38.024709 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:38.024751 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:38.025096 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:38.025585 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:38.026313 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:38.026504 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:38.026555 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:38.029082 (+ 2527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.194:52175: BlockingRecv error: recv got EOF from 127.0.196.194:52175 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 25 19:05:38 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:38 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:38.169582 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:38.164398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.195:51407)
0925 19:05:38.164556 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:38.164566 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:38.164628 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:38.164768 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:38.164769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:38.164963 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:38.165010 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:38.165012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:38.165741 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:38.165742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:38.166505 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:38.166508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:38.166975 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:38.166976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:38.167025 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:38.167561 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:38.167571 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:38.169519 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":100,"threads_started":1}
W20260925 19:05:38.169701 9978 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:38.164474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:51407 (local address 127.0.196.254:34733)
0925 19:05:38.164562 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:38.164564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:38.164636 (+ 72us) server_negotiation.cc:407] Connection header received
0925 19:05:38.164670 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:38.164671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:38.164684 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:38.164760 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:38.165087 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:38.165643 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:38.166598 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:38.166884 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:38.166950 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:38.169625 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.195:51407: BlockingRecv error: recv got EOF from 127.0.196.195:51407 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 19:05:38 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:38 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:38.917860 9981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:38.908934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.193:51597)
0925 19:05:38.909061 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:38.909074 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:38.909157 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:38.909337 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:38.909338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:38.909577 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:38.909638 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:38.909641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:38.910294 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:38.910295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:38.910948 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:38.910951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:38.911367 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:38.911368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:38.911406 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:38.915812 (+ 4406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:38.915818 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:38.917779 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
W20260925 19:05:38.918087 9982 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:38.909034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:51597 (local address 127.0.196.254:34733)
0925 19:05:38.909174 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:38.909176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:38.909188 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:05:38.909244 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:38.909244 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:38.909257 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:38.909329 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:38.909716 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:38.910218 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:38.911051 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:38.911288 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:38.911353 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:38.917914 (+ 6561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.193:51597: BlockingRecv error: recv got EOF from 127.0.196.193:51597 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
Sep 25 19:05:39 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:39 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:39.034850 9983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:39.029589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.194:52601)
0925 19:05:39.029814 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:39.029830 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:39.029901 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:39.030079 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:39.030080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:39.030327 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:39.030388 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:39.030391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:39.031166 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:39.031167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:39.031794 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:39.031796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:39.032222 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:39.032223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:39.032275 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:39.032881 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:39.032886 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:39.034788 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":93,"threads_started":1}
W20260925 19:05:39.034991 9982 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:39.029802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:52601 (local address 127.0.196.254:34733)
0925 19:05:39.029864 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:39.029866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:39.029887 (+ 21us) server_negotiation.cc:407] Connection header received
0925 19:05:39.029979 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:39.029980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:39.029994 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:39.030058 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:39.030478 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:39.031062 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:39.031902 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:39.032121 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:39.032188 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:39.034908 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.194:52601: BlockingRecv error: recv got EOF from 127.0.196.194:52601 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 25 19:05:39 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:39 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.195@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:39.175338 9984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:39.170162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.195:49903)
0925 19:05:39.170368 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:39.170380 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:39.170478 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:39.170650 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:39.170652 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:39.170843 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:39.170910 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:39.170911 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:39.171659 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:39.171660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:39.172258 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:39.172260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:39.172626 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:39.172627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:39.172688 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:39.173279 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:39.173286 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:39.175276 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":115,"threads_started":1}
W20260925 19:05:39.175464 9982 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:39.170253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:49903 (local address 127.0.196.254:34733)
0925 19:05:39.170337 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:39.170340 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:39.170462 (+ 122us) server_negotiation.cc:407] Connection header received
0925 19:05:39.170538 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:39.170539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:39.170554 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:39.170650 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:39.170994 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:39.171575 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:39.172337 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:39.172538 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:39.172618 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:39.175367 (+ 2749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.195:49903: BlockingRecv error: recv got EOF from 127.0.196.195:49903 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 19:05:39 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:39 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.193@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:39.923792 9985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:39.918579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.193:45989)
0925 19:05:39.918796 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:39.918808 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:39.918866 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:39.919018 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:39.919018 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:39.919259 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:39.919310 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:39.919312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:39.919964 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:39.919965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:39.920553 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:39.920555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:39.921261 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:39.921262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:39.921331 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:39.921773 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:39.921778 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:39.923729 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":102,"threads_started":1}
W20260925 19:05:39.923964 9986 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:39.918673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:45989 (local address 127.0.196.254:34733)
0925 19:05:39.918860 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:39.918862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:39.918874 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:05:39.918932 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:39.918933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:39.918943 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:39.919013 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:39.919377 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:39.919884 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:39.920889 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:39.921156 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:39.921250 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:39.923851 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.193:45989: BlockingRecv error: recv got EOF from 127.0.196.193:45989 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
Sep 25 19:05:40 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:05:40 dist-test-slave-rn98 krb5kdc[8927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.196.194@KRBTEST.COM for oryx/127.0.196.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:05:40.040208 9987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:40.035488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:34733 (local address 127.0.196.194:34905)
0925 19:05:40.035603 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:40.035614 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:40.035696 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:40.035829 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:40.035829 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:40.035983 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:40.036031 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:40.036033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:40.036659 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:40.036660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:40.037279 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:40.037282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:40.037661 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:40.037662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:40.037704 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:40.038175 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:40.038179 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:40.040095 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.196.254:34733: Server oryx/127.0.196.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":66,"threads_started":1}
W20260925 19:05:40.040271 9986 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:40.035570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:34905 (local address 127.0.196.254:34733)
0925 19:05:40.035638 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:40.035640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:40.035678 (+ 38us) server_negotiation.cc:407] Connection header received
0925 19:05:40.035747 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:40.035747 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:40.035758 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:40.035814 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:40.036087 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:40.036585 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:40.037387 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:40.037588 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:40.037665 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:40.040184 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.194:34905: BlockingRecv error: recv got EOF from 127.0.196.194:34905 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:05:40.085523 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 9510
I20260925 19:05:40.090490 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 9662
I20260925 19:05:40.094835 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 9795
I20260925 19:05:40.099093 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 9440
2026-09-25T19:05:40Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17550 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T19:05:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:05:40Z Disabled control of system clock
I20260925 19:05:40.128403 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:33401
--webserver_interface=127.0.196.254
--webserver_port=0
--builtin_ntp_servers=127.0.196.212:46405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:33401
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 19:05:40.210551 9991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:40.210834 9991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:40.210857 9991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:40.212033 9991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:05:40.212075 9991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:40.212088 9991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:40.212100 9991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:05:40.212112 9991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:05:40.213439 9991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:46405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:33401
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:33401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:40.213660 9991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:40.213922 9991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:40.219379 9996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:40.219372 9997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:40.219364 9999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:40.219409 9991 server_base.cc:1034] running on GCE node
I20260925 19:05:40.219916 9991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:40.220220 9991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:40.221415 9991 hybrid_clock.cc:648] HybridClock initialized: now 1790363140221407 us; error 57 us; skew 500 ppm
I20260925 19:05:40.222924 9991 webserver.cc:460] Webserver started at http://127.0.196.254:37441/ using document root <none> and password file <none>
I20260925 19:05:40.223148 9991 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:40.223217 9991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:40.223330 9991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:40.224498 9991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "1c06a7ee555c4b0c98ebd9c4501e69e0"
format_stamp: "Formatted at 2026-09-25 19:05:40 on dist-test-slave-rn98"
server_key: "fa3b651eed442c5a5fabceb61eb9d461"
server_key_iv: "958b4bdccfc47a720aa91143f149c670"
server_key_version: "encryptionkey@0"
I20260925 19:05:40.224838 9991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "1c06a7ee555c4b0c98ebd9c4501e69e0"
format_stamp: "Formatted at 2026-09-25 19:05:40 on dist-test-slave-rn98"
server_key: "fa3b651eed442c5a5fabceb61eb9d461"
server_key_iv: "958b4bdccfc47a720aa91143f149c670"
server_key_version: "encryptionkey@0"
I20260925 19:05:40.226336 9991 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:05:40.227267 10005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:40.227464 9991 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:05:40.227559 9991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "1c06a7ee555c4b0c98ebd9c4501e69e0"
format_stamp: "Formatted at 2026-09-25 19:05:40 on dist-test-slave-rn98"
server_key: "fa3b651eed442c5a5fabceb61eb9d461"
server_key_iv: "958b4bdccfc47a720aa91143f149c670"
server_key_version: "encryptionkey@0"
I20260925 19:05:40.227623 9991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:40.245657 9991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:40.245962 9991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:40.246069 9991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:40.249955 9991 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:33401
I20260925 19:05:40.250023 10057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:33401 every 8 connection(s)
I20260925 19:05:40.250358 9991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:05:40.251389 10058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:40.253367 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 9991
I20260925 19:05:40.253441 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:05:40.253648 787 external_mini_cluster.cc:1428] Setting key d0114f34c76e06707581e49c3493fe4b
I20260925 19:05:40.253806 10058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c06a7ee555c4b0c98ebd9c4501e69e0: Bootstrap starting.
I20260925 19:05:40.254580 10058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c06a7ee555c4b0c98ebd9c4501e69e0: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:40.254925 10058 log.cc:824] T 00000000000000000000000000000000 P 1c06a7ee555c4b0c98ebd9c4501e69e0: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:40.255728 10058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c06a7ee555c4b0c98ebd9c4501e69e0: No bootstrap required, opened a new log
I20260925 19:05:40.258002 10058 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c06a7ee555c4b0c98ebd9c4501e69e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c06a7ee555c4b0c98ebd9c4501e69e0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33401 } }
I20260925 19:05:40.258183 10058 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c06a7ee555c4b0c98ebd9c4501e69e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:40.258225 10058 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c06a7ee555c4b0c98ebd9c4501e69e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c06a7ee555c4b0c98ebd9c4501e69e0, State: Initialized, Role: FOLLOWER
I20260925 19:05:40.258352 10058 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c06a7ee555c4b0c98ebd9c4501e69e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c06a7ee555c4b0c98ebd9c4501e69e0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33401 } }
I20260925 19:05:40.258451 10058 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c06a7ee555c4b0c98ebd9c4501e69e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:05:40.258524 10058 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c06a7ee555c4b0c98ebd9c4501e69e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:05:40.258574 10058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c06a7ee555c4b0c98ebd9c4501e69e0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:40.259178 10058 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c06a7ee555c4b0c98ebd9c4501e69e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c06a7ee555c4b0c98ebd9c4501e69e0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33401 } }
I20260925 19:05:40.259318 10058 leader_election.cc:302] T 00000000000000000000000000000000 P 1c06a7ee555c4b0c98ebd9c4501e69e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c06a7ee555c4b0c98ebd9c4501e69e0; no voters:
I20260925 19:05:40.259491 10058 leader_election.cc:288] T 00000000000000000000000000000000 P 1c06a7ee555c4b0c98ebd9c4501e69e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:05:40.259549 10063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c06a7ee555c4b0c98ebd9c4501e69e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:05:40.259707 10063 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c06a7ee555c4b0c98ebd9c4501e69e0 [term 1 LEADER]: Becoming Leader. State: Replica: 1c06a7ee555c4b0c98ebd9c4501e69e0, State: Running, Role: LEADER
I20260925 19:05:40.259807 10058 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c06a7ee555c4b0c98ebd9c4501e69e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:05:40.259845 10063 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c06a7ee555c4b0c98ebd9c4501e69e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c06a7ee555c4b0c98ebd9c4501e69e0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33401 } }
I20260925 19:05:40.260061 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:40.254217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46394 (local address 127.0.196.254:33401)
0925 19:05:40.254433 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:40.254442 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:40.254476 (+ 34us) server_negotiation.cc:407] Connection header received
0925 19:05:40.255161 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:40.255176 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:40.255248 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:40.255404 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:40.255569 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:40.256956 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:40.257784 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:40.258396 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:40.258528 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:40.259275 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:40.259279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:40.259279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:05:40.259500 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:40.259544 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:05:40.259563 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:40.259629 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:40.259742 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 19:05:40.259834 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
I20260925 19:05:40.260655 10065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c06a7ee555c4b0c98ebd9c4501e69e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c06a7ee555c4b0c98ebd9c4501e69e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c06a7ee555c4b0c98ebd9c4501e69e0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33401 } } }
I20260925 19:05:40.260748 10065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c06a7ee555c4b0c98ebd9c4501e69e0 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:40.260907 10064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c06a7ee555c4b0c98ebd9c4501e69e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c06a7ee555c4b0c98ebd9c4501e69e0. Latest consensus state: current_term: 1 leader_uuid: "1c06a7ee555c4b0c98ebd9c4501e69e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c06a7ee555c4b0c98ebd9c4501e69e0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33401 } } }
I20260925 19:05:40.260941 10072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:05:40.260973 10064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c06a7ee555c4b0c98ebd9c4501e69e0 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:40.261447 10072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:05:40.263025 10072 catalog_manager.cc:1347] Generated new cluster ID: 573dee52f852456b83c4c6bc4713e43f
I20260925 19:05:40.263077 10072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:05:40.268827 10072 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:05:40.269351 10072 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:05:40.282688 10072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c06a7ee555c4b0c98ebd9c4501e69e0: Generated new TSK 0
I20260925 19:05:40.282931 10072 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:05:40.291074 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:0
--local_ip_for_outbound_sockets=127.0.196.193
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:33401
--builtin_ntp_servers=127.0.196.212:46405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 19:05:40.374001 10082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:40.374224 10082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:40.374248 10082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:40.375510 10082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:40.375553 10082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:40.375581 10082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:05:40.376964 10082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:46405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:33401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:40.377224 10082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:40.377525 10082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:40.382664 10088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:40.382711 10087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:40.382835 10090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:40.383127 10082 server_base.cc:1034] running on GCE node
I20260925 19:05:40.383342 10082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:40.383599 10082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:40.384766 10082 hybrid_clock.cc:648] HybridClock initialized: now 1790363140384744 us; error 32 us; skew 500 ppm
I20260925 19:05:40.386411 10082 webserver.cc:460] Webserver started at http://127.0.196.193:43719/ using document root <none> and password file <none>
I20260925 19:05:40.386729 10082 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:40.386793 10082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:40.386926 10082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:40.388319 10082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/data/instance:
uuid: "0c1d63d38c4d4e95b904f93e8de2c566"
format_stamp: "Formatted at 2026-09-25 19:05:40 on dist-test-slave-rn98"
server_key: "c439e3f339ce7fed5ebcca5a21db8cbf"
server_key_iv: "d5dec051afa90d95f1e734a46b042793"
server_key_version: "encryptionkey@0"
I20260925 19:05:40.388772 10082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/wal/instance:
uuid: "0c1d63d38c4d4e95b904f93e8de2c566"
format_stamp: "Formatted at 2026-09-25 19:05:40 on dist-test-slave-rn98"
server_key: "c439e3f339ce7fed5ebcca5a21db8cbf"
server_key_iv: "d5dec051afa90d95f1e734a46b042793"
server_key_version: "encryptionkey@0"
I20260925 19:05:40.390244 10082 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:05:40.391219 10096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:40.391423 10082 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:05:40.391526 10082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "0c1d63d38c4d4e95b904f93e8de2c566"
format_stamp: "Formatted at 2026-09-25 19:05:40 on dist-test-slave-rn98"
server_key: "c439e3f339ce7fed5ebcca5a21db8cbf"
server_key_iv: "d5dec051afa90d95f1e734a46b042793"
server_key_version: "encryptionkey@0"
I20260925 19:05:40.391610 10082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:40.399436 10082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:40.399783 10082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:40.399931 10082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:40.400157 10082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:40.400483 10082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:40.400533 10082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:40.400573 10082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:40.400599 10082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:40.406523 10082 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:35049
I20260925 19:05:40.406592 10209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:35049 every 8 connection(s)
I20260925 19:05:40.406982 10082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
I20260925 19:05:40.412345 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:40.407783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:59271 (local address 127.0.196.254:33401)
0925 19:05:40.407851 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:40.407853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:40.408089 (+ 236us) server_negotiation.cc:407] Connection header received
0925 19:05:40.408612 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:40.408613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:40.408629 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:40.408695 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:40.409348 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:40.409857 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:40.410901 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:40.411113 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:40.411205 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:40.412051 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:40.412053 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:40.412053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:05:40.412058 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:40.412073 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:05:40.412077 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:40.412144 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:40.412234 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 19:05:40.412262 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 19:05:40.412582 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:40.407776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33401 (local address 127.0.196.193:59271)
0925 19:05:40.407969 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:40.408002 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:40.408509 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:40.408770 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:40.408773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:40.408783 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:40.409256 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:40.409260 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:40.409950 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:40.409951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:40.410810 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:40.410812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:40.411214 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:40.411215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:40.411349 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:40.411909 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:05:40.411909 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:40.411938 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:05:40.411961 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:40.412144 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:40.412150 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:40.412153 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:05:40.412236 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 19:05:40.412338 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
I20260925 19:05:40.413094 10210 heartbeater.cc:347] Connected to a master server at 127.0.196.254:33401
I20260925 19:05:40.413182 10210 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:40.413327 10210 heartbeater.cc:511] Master 127.0.196.254:33401 requested a full tablet report, sending...
I20260925 19:05:40.413820 10022 ts_manager.cc:194] Registered new tserver with Master: 0c1d63d38c4d4e95b904f93e8de2c566 (127.0.196.193:35049)
I20260925 19:05:40.414449 10022 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.196.193:59271
I20260925 19:05:40.416370 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 10082
I20260925 19:05:40.416438 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/wal/instance
I20260925 19:05:40.416594 787 external_mini_cluster.cc:1428] Setting key ee13c9d913e455c77496e0700bf1a695
I20260925 19:05:40.417460 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:0
--local_ip_for_outbound_sockets=127.0.196.194
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:33401
--builtin_ntp_servers=127.0.196.212:46405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 19:05:40.498327 10213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:40.498535 10213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:40.498558 10213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:40.499763 10213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:40.499807 10213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:40.499832 10213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:05:40.501096 10213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:46405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:33401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:40.501289 10213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:40.501537 10213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:40.506461 10219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:40.506413 10221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:40.506428 10218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:40.506592 10213 server_base.cc:1034] running on GCE node
I20260925 19:05:40.506917 10213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:40.507141 10213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:40.508293 10213 hybrid_clock.cc:648] HybridClock initialized: now 1790363140508279 us; error 29 us; skew 500 ppm
I20260925 19:05:40.509596 10213 webserver.cc:460] Webserver started at http://127.0.196.194:39597/ using document root <none> and password file <none>
I20260925 19:05:40.509851 10213 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:40.509900 10213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:40.510028 10213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:40.511194 10213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/data/instance:
uuid: "77edc0aedafd4da99a826c22573e4d1c"
format_stamp: "Formatted at 2026-09-25 19:05:40 on dist-test-slave-rn98"
server_key: "690caae7e8587e9fc76e1aace541d1c9"
server_key_iv: "aac06f5912a093ad66e45bea17cf146d"
server_key_version: "encryptionkey@0"
I20260925 19:05:40.511559 10213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/wal/instance:
uuid: "77edc0aedafd4da99a826c22573e4d1c"
format_stamp: "Formatted at 2026-09-25 19:05:40 on dist-test-slave-rn98"
server_key: "690caae7e8587e9fc76e1aace541d1c9"
server_key_iv: "aac06f5912a093ad66e45bea17cf146d"
server_key_version: "encryptionkey@0"
I20260925 19:05:40.512907 10213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:05:40.513661 10227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:40.513864 10213 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:40.513940 10213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "77edc0aedafd4da99a826c22573e4d1c"
format_stamp: "Formatted at 2026-09-25 19:05:40 on dist-test-slave-rn98"
server_key: "690caae7e8587e9fc76e1aace541d1c9"
server_key_iv: "aac06f5912a093ad66e45bea17cf146d"
server_key_version: "encryptionkey@0"
I20260925 19:05:40.513996 10213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:40.530097 10213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:40.530391 10213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:40.530496 10213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:40.530751 10213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:40.531049 10213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:40.531082 10213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:40.531105 10213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:40.531118 10213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:40.536572 10213 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:46063
I20260925 19:05:40.536665 10340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:46063 every 8 connection(s)
I20260925 19:05:40.536978 10213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
I20260925 19:05:40.542253 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:40.537691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:53001 (local address 127.0.196.254:33401)
0925 19:05:40.537764 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:40.537766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:40.538013 (+ 247us) server_negotiation.cc:407] Connection header received
0925 19:05:40.538554 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:40.538555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:40.538567 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:40.538653 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:40.539344 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:40.539864 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:40.540855 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:40.541059 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:40.541132 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:40.541953 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:40.541954 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:40.541954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:05:40.541959 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:40.541972 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:05:40.541981 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:40.542037 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:40.542132 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 19:05:40.542156 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 19:05:40.542469 10343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:40.537683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33401 (local address 127.0.196.194:53001)
0925 19:05:40.537889 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:40.537923 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:40.538446 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:40.538731 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:40.538735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:40.538746 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:40.539255 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:40.539259 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:40.539964 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:40.539965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:40.540763 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:40.540765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:40.541139 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:40.541140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:40.541279 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:40.541803 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:05:40.541804 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:40.541834 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:05:40.541856 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:40.542037 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:40.542042 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:40.542045 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:05:40.542122 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 19:05:40.542222 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
I20260925 19:05:40.542686 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 10213
I20260925 19:05:40.542779 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/wal/instance
I20260925 19:05:40.542915 10341 heartbeater.cc:347] Connected to a master server at 127.0.196.254:33401
I20260925 19:05:40.542968 787 external_mini_cluster.cc:1428] Setting key 432680cdc27254b5ed443086cf6bfbe3
I20260925 19:05:40.543004 10341 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:40.543222 10341 heartbeater.cc:511] Master 127.0.196.254:33401 requested a full tablet report, sending...
I20260925 19:05:40.543587 10022 ts_manager.cc:194] Registered new tserver with Master: 77edc0aedafd4da99a826c22573e4d1c (127.0.196.194:46063)
I20260925 19:05:40.543994 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:0
--local_ip_for_outbound_sockets=127.0.196.195
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:33401
--builtin_ntp_servers=127.0.196.212:46405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 19:05:40.544100 10022 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.196.194:53001
W20260925 19:05:40.628019 10344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:40.628249 10344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:40.628270 10344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:40.629439 10344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:40.629479 10344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:40.629503 10344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:05:40.630851 10344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:46405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:33401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:40.631043 10344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:40.631291 10344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:40.636312 10349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:40.636313 10350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:40.636525 10344 server_base.cc:1034] running on GCE node
W20260925 19:05:40.636387 10352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:40.636821 10344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:40.637094 10344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:40.638257 10344 hybrid_clock.cc:648] HybridClock initialized: now 1790363140638243 us; error 27 us; skew 500 ppm
I20260925 19:05:40.639739 10344 webserver.cc:460] Webserver started at http://127.0.196.195:38337/ using document root <none> and password file <none>
I20260925 19:05:40.639989 10344 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:40.640056 10344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:40.640192 10344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:40.641290 10344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/data/instance:
uuid: "9155cc96c2b443c9a4a18b5acf2daa0e"
format_stamp: "Formatted at 2026-09-25 19:05:40 on dist-test-slave-rn98"
server_key: "8c15710995a4a8d2878548e90245697e"
server_key_iv: "59964a694f845cea42afe06d25eca263"
server_key_version: "encryptionkey@0"
I20260925 19:05:40.641651 10344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/wal/instance:
uuid: "9155cc96c2b443c9a4a18b5acf2daa0e"
format_stamp: "Formatted at 2026-09-25 19:05:40 on dist-test-slave-rn98"
server_key: "8c15710995a4a8d2878548e90245697e"
server_key_iv: "59964a694f845cea42afe06d25eca263"
server_key_version: "encryptionkey@0"
I20260925 19:05:40.643070 10344 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 19:05:40.643849 10358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:40.644024 10344 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:05:40.644120 10344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "9155cc96c2b443c9a4a18b5acf2daa0e"
format_stamp: "Formatted at 2026-09-25 19:05:40 on dist-test-slave-rn98"
server_key: "8c15710995a4a8d2878548e90245697e"
server_key_iv: "59964a694f845cea42afe06d25eca263"
server_key_version: "encryptionkey@0"
I20260925 19:05:40.644197 10344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:40.656312 10344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:40.656641 10344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:40.656791 10344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:40.657023 10344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:40.657362 10344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:40.657413 10344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:40.657455 10344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:40.657481 10344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:40.663888 10344 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:41453
I20260925 19:05:40.663960 10471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:41453 every 8 connection(s)
I20260925 19:05:40.664321 10344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
I20260925 19:05:40.669386 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 10344
I20260925 19:05:40.669481 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/wal/instance
I20260925 19:05:40.669719 787 external_mini_cluster.cc:1428] Setting key a63f5b23bf8e82f8adaf62c3286f4354
I20260925 19:05:40.669939 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:40.665137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:50173 (local address 127.0.196.254:33401)
0925 19:05:40.665195 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:40.665198 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:40.665534 (+ 336us) server_negotiation.cc:407] Connection header received
0925 19:05:40.666055 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:40.666056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:40.666066 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:40.666121 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:40.666897 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:40.667390 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:40.668405 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:40.668598 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:40.668674 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:40.669616 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:40.669617 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:40.669618 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:05:40.669623 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:40.669638 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:05:40.669643 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:40.669709 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:40.669825 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 19:05:40.669855 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 19:05:40.670176 10474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:40.665141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33401 (local address 127.0.196.195:50173)
0925 19:05:40.665408 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:40.665456 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:40.665968 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:40.666180 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:40.666183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:40.666192 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:40.666790 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:40.666797 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:40.667472 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:40.667473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:40.668305 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:40.668307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:40.668685 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:40.668686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:40.668801 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:40.669426 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:05:40.669426 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:40.669482 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:05:40.669514 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:40.669708 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:40.669715 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:40.669718 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:05:40.669820 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 19:05:40.669929 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":97,"threads_started":1}
I20260925 19:05:40.670704 10472 heartbeater.cc:347] Connected to a master server at 127.0.196.254:33401
I20260925 19:05:40.670821 10472 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:40.671025 10472 heartbeater.cc:511] Master 127.0.196.254:33401 requested a full tablet report, sending...
I20260925 19:05:40.671459 10021 ts_manager.cc:194] Registered new tserver with Master: 9155cc96c2b443c9a4a18b5acf2daa0e (127.0.196.195:41453)
I20260925 19:05:40.671878 10021 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.196.195:50173
I20260925 19:05:40.682068 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 19:05:40.684137 10061 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:40.683527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46406 (local address 127.0.196.254:33401)
0925 19:05:40.683602 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:40.683618 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:40.683742 (+ 124us) server_negotiation.cc:407] Connection header received
0925 19:05:40.683829 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:40.683830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:40.683840 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:40.683889 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:40.684051 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46406: BlockingRecv error: recv got EOF from 127.0.0.1:46406 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 19:05:40.684170 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 10082
I20260925 19:05:40.689106 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 10213
I20260925 19:05:40.693624 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 10344
I20260925 19:05:40.697824 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 9991
2026-09-25T19:05:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (597 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T19:05:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:05:40Z Disabled control of system clock
I20260925 19:05:40.724409 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:42375
--webserver_interface=127.0.196.254
--webserver_port=0
--builtin_ntp_servers=127.0.196.212:45375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:42375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 19:05:40.804759 10483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:40.804971 10483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:40.804992 10483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:40.806084 10483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:05:40.806120 10483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:40.806133 10483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:40.806145 10483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:05:40.806157 10483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:05:40.807576 10483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:45375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:42375
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:42375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:40.807787 10483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:40.808029 10483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:40.812942 10488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:40.812906 10491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:40.812925 10489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:40.813093 10483 server_base.cc:1034] running on GCE node
I20260925 19:05:40.813354 10483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:40.813679 10483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:40.814841 10483 hybrid_clock.cc:648] HybridClock initialized: now 1790363140814821 us; error 37 us; skew 500 ppm
I20260925 19:05:40.816161 10483 webserver.cc:460] Webserver started at http://127.0.196.254:36309/ using document root <none> and password file <none>
I20260925 19:05:40.816447 10483 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:40.816522 10483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:40.816658 10483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:40.817716 10483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "aa8f5889407a4b14a82ae9d7b0ffca8b"
format_stamp: "Formatted at 2026-09-25 19:05:40 on dist-test-slave-rn98"
server_key: "d007f4d902271e81cdbfb3c83b8d50cf"
server_key_iv: "0e97080814bf719a04a9c409f2749d5f"
server_key_version: "encryptionkey@0"
I20260925 19:05:40.818090 10483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "aa8f5889407a4b14a82ae9d7b0ffca8b"
format_stamp: "Formatted at 2026-09-25 19:05:40 on dist-test-slave-rn98"
server_key: "d007f4d902271e81cdbfb3c83b8d50cf"
server_key_iv: "0e97080814bf719a04a9c409f2749d5f"
server_key_version: "encryptionkey@0"
I20260925 19:05:40.819478 10483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:05:40.820237 10497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:40.820408 10483 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:05:40.820504 10483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "aa8f5889407a4b14a82ae9d7b0ffca8b"
format_stamp: "Formatted at 2026-09-25 19:05:40 on dist-test-slave-rn98"
server_key: "d007f4d902271e81cdbfb3c83b8d50cf"
server_key_iv: "0e97080814bf719a04a9c409f2749d5f"
server_key_version: "encryptionkey@0"
I20260925 19:05:40.820580 10483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:40.823975 10483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:40.824241 10483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:40.824374 10483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:40.828269 10483 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:42375
I20260925 19:05:40.828302 10549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:42375 every 8 connection(s)
I20260925 19:05:40.828657 10483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:05:40.829290 10550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:40.832052 10550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa8f5889407a4b14a82ae9d7b0ffca8b: Bootstrap starting.
I20260925 19:05:40.832580 10550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa8f5889407a4b14a82ae9d7b0ffca8b: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:40.832816 10550 log.cc:824] T 00000000000000000000000000000000 P aa8f5889407a4b14a82ae9d7b0ffca8b: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:40.833498 10550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa8f5889407a4b14a82ae9d7b0ffca8b: No bootstrap required, opened a new log
I20260925 19:05:40.835131 10550 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa8f5889407a4b14a82ae9d7b0ffca8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8f5889407a4b14a82ae9d7b0ffca8b" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 42375 } }
I20260925 19:05:40.835304 10550 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa8f5889407a4b14a82ae9d7b0ffca8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:40.835328 10550 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa8f5889407a4b14a82ae9d7b0ffca8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa8f5889407a4b14a82ae9d7b0ffca8b, State: Initialized, Role: FOLLOWER
I20260925 19:05:40.835462 10550 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa8f5889407a4b14a82ae9d7b0ffca8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8f5889407a4b14a82ae9d7b0ffca8b" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 42375 } }
I20260925 19:05:40.835539 10550 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa8f5889407a4b14a82ae9d7b0ffca8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:05:40.835562 10550 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa8f5889407a4b14a82ae9d7b0ffca8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:05:40.835590 10550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa8f5889407a4b14a82ae9d7b0ffca8b [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:40.836079 10550 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa8f5889407a4b14a82ae9d7b0ffca8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8f5889407a4b14a82ae9d7b0ffca8b" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 42375 } }
I20260925 19:05:40.836169 10550 leader_election.cc:302] T 00000000000000000000000000000000 P aa8f5889407a4b14a82ae9d7b0ffca8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa8f5889407a4b14a82ae9d7b0ffca8b; no voters:
I20260925 19:05:40.836293 10550 leader_election.cc:288] T 00000000000000000000000000000000 P aa8f5889407a4b14a82ae9d7b0ffca8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:05:40.836417 10553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa8f5889407a4b14a82ae9d7b0ffca8b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:05:40.836537 10550 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa8f5889407a4b14a82ae9d7b0ffca8b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:05:40.836580 10553 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa8f5889407a4b14a82ae9d7b0ffca8b [term 1 LEADER]: Becoming Leader. State: Replica: aa8f5889407a4b14a82ae9d7b0ffca8b, State: Running, Role: LEADER
I20260925 19:05:40.836772 10553 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa8f5889407a4b14a82ae9d7b0ffca8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8f5889407a4b14a82ae9d7b0ffca8b" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 42375 } }
I20260925 19:05:40.837172 10554 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa8f5889407a4b14a82ae9d7b0ffca8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa8f5889407a4b14a82ae9d7b0ffca8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8f5889407a4b14a82ae9d7b0ffca8b" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 42375 } } }
I20260925 19:05:40.837337 10554 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa8f5889407a4b14a82ae9d7b0ffca8b [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:40.837392 10555 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa8f5889407a4b14a82ae9d7b0ffca8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa8f5889407a4b14a82ae9d7b0ffca8b. Latest consensus state: current_term: 1 leader_uuid: "aa8f5889407a4b14a82ae9d7b0ffca8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8f5889407a4b14a82ae9d7b0ffca8b" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 42375 } } }
I20260925 19:05:40.837530 10555 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa8f5889407a4b14a82ae9d7b0ffca8b [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:40.837836 10562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:05:40.838335 10562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:05:40.838737 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 10483
I20260925 19:05:40.838835 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:05:40.839013 787 external_mini_cluster.cc:1428] Setting key fa2ddef3280d34abe79599e211a77ae5
I20260925 19:05:40.840221 10562 catalog_manager.cc:1347] Generated new cluster ID: 0e33068162fa4596b72a056bc79d93d7
I20260925 19:05:40.840276 10562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:05:40.841778 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:40.839873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44146 (local address 127.0.196.254:42375)
0925 19:05:40.840041 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:40.840049 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:40.840071 (+ 22us) server_negotiation.cc:407] Connection header received
0925 19:05:40.840202 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:40.840213 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:40.840247 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:40.840396 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:40.841333 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:40.841337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:40.841337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:05:40.841341 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:40.841387 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:05:40.841397 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:40.841467 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:40.841540 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 19:05:40.841605 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260925 19:05:40.869328 10562 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:05:40.869541 10562 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:05:40.872622 10562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa8f5889407a4b14a82ae9d7b0ffca8b: Generated new TSK 0
I20260925 19:05:40.872845 10562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:05:40.878069 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:0
--local_ip_for_outbound_sockets=127.0.196.193
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:42375
--builtin_ntp_servers=127.0.196.212:45375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 19:05:40.964820 10574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:40.965034 10574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:40.965057 10574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:40.966373 10574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:40.966413 10574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:40.966439 10574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:05:40.967842 10574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:45375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:42375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:40.968099 10574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:40.968410 10574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:40.973425 10582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:40.973425 10579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:40.973765 10574 server_base.cc:1034] running on GCE node
W20260925 19:05:40.973506 10580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:40.974082 10574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:40.974359 10574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:40.975520 10574 hybrid_clock.cc:648] HybridClock initialized: now 1790363140975485 us; error 57 us; skew 500 ppm
I20260925 19:05:40.976832 10574 webserver.cc:460] Webserver started at http://127.0.196.193:35031/ using document root <none> and password file <none>
I20260925 19:05:40.977092 10574 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:40.977138 10574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:40.977296 10574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:40.978375 10574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/data/instance:
uuid: "761b74dfe3524c7c8df9f2552d757498"
format_stamp: "Formatted at 2026-09-25 19:05:40 on dist-test-slave-rn98"
server_key: "c9469c848e5b27150f924f4ae02ef384"
server_key_iv: "a5a1c81b86fcb3ba8ac80bebadd34f7a"
server_key_version: "encryptionkey@0"
I20260925 19:05:40.978739 10574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/wal/instance:
uuid: "761b74dfe3524c7c8df9f2552d757498"
format_stamp: "Formatted at 2026-09-25 19:05:40 on dist-test-slave-rn98"
server_key: "c9469c848e5b27150f924f4ae02ef384"
server_key_iv: "a5a1c81b86fcb3ba8ac80bebadd34f7a"
server_key_version: "encryptionkey@0"
I20260925 19:05:40.980050 10574 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:40.980826 10588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:40.981010 10574 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:40.981104 10574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "761b74dfe3524c7c8df9f2552d757498"
format_stamp: "Formatted at 2026-09-25 19:05:40 on dist-test-slave-rn98"
server_key: "c9469c848e5b27150f924f4ae02ef384"
server_key_iv: "a5a1c81b86fcb3ba8ac80bebadd34f7a"
server_key_version: "encryptionkey@0"
I20260925 19:05:40.981197 10574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:40.984743 10574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:40.985044 10574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:40.985185 10574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:40.985383 10574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:40.985689 10574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:40.985739 10574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:40.985777 10574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:40.985803 10574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:40.991652 10574 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:34113
I20260925 19:05:40.991706 10701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:34113 every 8 connection(s)
I20260925 19:05:40.992076 10574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
I20260925 19:05:40.993391 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 10574
I20260925 19:05:40.993495 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-0/wal/instance
I20260925 19:05:40.993680 787 external_mini_cluster.cc:1428] Setting key e36cb6aea4710d3f25b86560ca04d9ae
I20260925 19:05:40.994728 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:0
--local_ip_for_outbound_sockets=127.0.196.194
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:42375
--builtin_ntp_servers=127.0.196.212:45375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 19:05:40.995584 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:40.992832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:44713 (local address 127.0.196.254:42375)
0925 19:05:40.992903 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:40.992905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:40.993247 (+ 342us) server_negotiation.cc:407] Connection header received
0925 19:05:40.993847 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:40.993849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:40.993857 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:40.993902 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:40.995318 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:40.995319 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:40.995319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:05:40.995324 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:40.995343 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:05:40.995349 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:40.995419 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:40.995468 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 19:05:40.995484 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 19:05:40.995725 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:40.992822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:42375 (local address 127.0.196.193:44713)
0925 19:05:40.993010 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:40.993043 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:40.993725 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:40.993989 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:40.994008 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:40.994019 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:40.995161 (+ 1142us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:05:40.995161 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:40.995193 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:05:40.995231 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:40.995418 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:40.995421 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:40.995423 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:05:40.995474 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 19:05:40.995522 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":88,"threads_started":1}
I20260925 19:05:40.996194 10702 heartbeater.cc:347] Connected to a master server at 127.0.196.254:42375
I20260925 19:05:40.996286 10702 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:40.996431 10702 heartbeater.cc:511] Master 127.0.196.254:42375 requested a full tablet report, sending...
I20260925 19:05:40.997018 10514 ts_manager.cc:194] Registered new tserver with Master: 761b74dfe3524c7c8df9f2552d757498 (127.0.196.193:34113)
W20260925 19:05:41.076707 10705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:41.076938 10705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:41.076961 10705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:41.078145 10705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:41.078198 10705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:41.078225 10705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:05:41.079778 10705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:45375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:42375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:41.080201 10705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:41.080634 10705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:41.086791 10710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:41.087263 10711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:41.086777 10713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:41.087937 10705 server_base.cc:1034] running on GCE node
I20260925 19:05:41.088187 10705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:41.088490 10705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:41.089684 10705 hybrid_clock.cc:648] HybridClock initialized: now 1790363141089672 us; error 27 us; skew 500 ppm
I20260925 19:05:41.091156 10705 webserver.cc:460] Webserver started at http://127.0.196.194:43083/ using document root <none> and password file <none>
I20260925 19:05:41.091440 10705 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:41.091514 10705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:41.091688 10705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:41.092904 10705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/data/instance:
uuid: "f4750b412acc4eafa7701ce5397d0c0d"
format_stamp: "Formatted at 2026-09-25 19:05:41 on dist-test-slave-rn98"
server_key: "2934b86f491ffb80d5b43ce62fe0fb51"
server_key_iv: "714ca2d229a770a7572561399faa5f64"
server_key_version: "encryptionkey@0"
I20260925 19:05:41.093246 10705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/wal/instance:
uuid: "f4750b412acc4eafa7701ce5397d0c0d"
format_stamp: "Formatted at 2026-09-25 19:05:41 on dist-test-slave-rn98"
server_key: "2934b86f491ffb80d5b43ce62fe0fb51"
server_key_iv: "714ca2d229a770a7572561399faa5f64"
server_key_version: "encryptionkey@0"
I20260925 19:05:41.094796 10705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:05:41.095543 10719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:41.095713 10705 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:05:41.095801 10705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "f4750b412acc4eafa7701ce5397d0c0d"
format_stamp: "Formatted at 2026-09-25 19:05:41 on dist-test-slave-rn98"
server_key: "2934b86f491ffb80d5b43ce62fe0fb51"
server_key_iv: "714ca2d229a770a7572561399faa5f64"
server_key_version: "encryptionkey@0"
I20260925 19:05:41.095875 10705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:41.099241 10705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:41.099463 10705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:41.099560 10705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:41.099762 10705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:41.100029 10705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:41.100059 10705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:41.100080 10705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:41.100097 10705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:41.105417 10705 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:33893
I20260925 19:05:41.105556 10832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:33893 every 8 connection(s)
I20260925 19:05:41.105808 10705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
I20260925 19:05:41.108901 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:41.106534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:41759 (local address 127.0.196.254:42375)
0925 19:05:41.106612 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:41.106647 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:41.106946 (+ 299us) server_negotiation.cc:407] Connection header received
0925 19:05:41.107447 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:41.107448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:41.107453 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:41.107506 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:41.108687 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:41.108688 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:41.108689 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:05:41.108692 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:41.108705 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:05:41.108710 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:41.108759 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:41.108835 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 19:05:41.108846 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 19:05:41.109119 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:41.106494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:42375 (local address 127.0.196.194:41759)
0925 19:05:41.106825 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:41.106861 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:41.107361 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:41.107564 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:41.107567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:41.107576 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:41.108536 (+ 960us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:05:41.108537 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:41.108568 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:05:41.108599 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:41.108763 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:41.108766 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:41.108768 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:05:41.108824 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 19:05:41.108875 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":100,"threads_started":1}
I20260925 19:05:41.109557 10833 heartbeater.cc:347] Connected to a master server at 127.0.196.254:42375
I20260925 19:05:41.109748 10833 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:41.109918 10833 heartbeater.cc:511] Master 127.0.196.254:42375 requested a full tablet report, sending...
I20260925 19:05:41.109992 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 10705
I20260925 19:05:41.110082 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-1/wal/instance
I20260925 19:05:41.110297 787 external_mini_cluster.cc:1428] Setting key 031e92456335d1aaff9e16cc05cad17b
I20260925 19:05:41.110321 10514 ts_manager.cc:194] Registered new tserver with Master: f4750b412acc4eafa7701ce5397d0c0d (127.0.196.194:33893)
I20260925 19:05:41.111212 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:0
--local_ip_for_outbound_sockets=127.0.196.195
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:42375
--builtin_ntp_servers=127.0.196.212:45375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 19:05:41.192355 10836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:41.192566 10836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:41.192588 10836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:41.193780 10836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:41.193819 10836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:41.193845 10836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:05:41.195289 10836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:45375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:42375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:41.195533 10836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:41.195850 10836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:41.200907 10844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:41.200902 10841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:41.200902 10842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:41.201205 10836 server_base.cc:1034] running on GCE node
I20260925 19:05:41.201439 10836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:41.201686 10836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:41.202831 10836 hybrid_clock.cc:648] HybridClock initialized: now 1790363141202816 us; error 35 us; skew 500 ppm
I20260925 19:05:41.204200 10836 webserver.cc:460] Webserver started at http://127.0.196.195:40705/ using document root <none> and password file <none>
I20260925 19:05:41.204466 10836 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:41.204516 10836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:41.204731 10836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:41.205946 10836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/data/instance:
uuid: "d3ddb6a9907e409bb0b8ce1e903cca07"
format_stamp: "Formatted at 2026-09-25 19:05:41 on dist-test-slave-rn98"
server_key: "7e8487cb5e75356cb278d33c33730d79"
server_key_iv: "7fc36ea405967744878e1159e0f7d625"
server_key_version: "encryptionkey@0"
I20260925 19:05:41.206336 10836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/wal/instance:
uuid: "d3ddb6a9907e409bb0b8ce1e903cca07"
format_stamp: "Formatted at 2026-09-25 19:05:41 on dist-test-slave-rn98"
server_key: "7e8487cb5e75356cb278d33c33730d79"
server_key_iv: "7fc36ea405967744878e1159e0f7d625"
server_key_version: "encryptionkey@0"
I20260925 19:05:41.207794 10836 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:05:41.208585 10850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:41.208809 10836 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:41.208906 10836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "d3ddb6a9907e409bb0b8ce1e903cca07"
format_stamp: "Formatted at 2026-09-25 19:05:41 on dist-test-slave-rn98"
server_key: "7e8487cb5e75356cb278d33c33730d79"
server_key_iv: "7fc36ea405967744878e1159e0f7d625"
server_key_version: "encryptionkey@0"
I20260925 19:05:41.208981 10836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:41.212780 10836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:41.213068 10836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:41.213217 10836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:41.213442 10836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:41.213769 10836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:41.213824 10836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:41.213869 10836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:41.213899 10836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:41.220320 10836 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:40773
I20260925 19:05:41.220367 10963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:40773 every 8 connection(s)
I20260925 19:05:41.220727 10836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
I20260925 19:05:41.223917 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:41.221519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:45167 (local address 127.0.196.254:42375)
0925 19:05:41.221594 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:41.221596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:41.221847 (+ 251us) server_negotiation.cc:407] Connection header received
0925 19:05:41.222330 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:41.222332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:41.222337 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:41.222400 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:41.223642 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:41.223643 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:41.223643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:05:41.223648 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:41.223663 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:05:41.223675 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:41.223727 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:41.223846 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 19:05:41.223860 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 19:05:41.224141 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:41.221471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:42375 (local address 127.0.196.195:45167)
0925 19:05:41.221697 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:41.221733 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:41.222228 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:41.222472 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:41.222476 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:41.222485 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:41.223493 (+ 1008us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:05:41.223494 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:41.223523 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:05:41.223552 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:41.223769 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:41.223773 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:41.223776 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:05:41.223847 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 19:05:41.223920 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":117,"threads_started":1}
I20260925 19:05:41.224537 10964 heartbeater.cc:347] Connected to a master server at 127.0.196.254:42375
I20260925 19:05:41.224627 10964 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:41.224795 10964 heartbeater.cc:511] Master 127.0.196.254:42375 requested a full tablet report, sending...
I20260925 19:05:41.225111 10514 ts_manager.cc:194] Registered new tserver with Master: d3ddb6a9907e409bb0b8ce1e903cca07 (127.0.196.195:40773)
I20260925 19:05:41.225730 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 10836
I20260925 19:05:41.225822 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790363087086945-787-0/minicluster-data/ts-2/wal/instance
I20260925 19:05:41.226001 787 external_mini_cluster.cc:1428] Setting key 54aeade1745f1f469852f91619592753
I20260925 19:05:41.227056 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 19:05:41.228910 10573 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:05:41.228372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44162 (local address 127.0.196.254:42375)
0925 19:05:41.228440 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:41.228442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:41.228554 (+ 112us) server_negotiation.cc:407] Connection header received
0925 19:05:41.228629 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:41.228629 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:41.228635 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:41.228708 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:41.228841 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44162: BlockingRecv error: recv got EOF from 127.0.0.1:44162 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:05:41.229005 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 10574
I20260925 19:05:41.235080 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 10705
I20260925 19:05:41.240193 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 10836
I20260925 19:05:41.244925 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 10483
2026-09-25T19:05:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (547 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:05:41 dist-test-slave-rn98 krb5kdc[10974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:05:41 dist-test-slave-rn98 krb5kdc[10974](info): set up 2 sockets
Sep 25 19:05:41 dist-test-slave-rn98 krb5kdc[10974](info): commencing operation
krb5kdc: starting...
W20260925 19:05:42.291757 787 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:05:42 dist-test-slave-rn98 krb5kdc[10974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:05:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:05:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:05:42.413934 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:36145
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:38351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:36145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:42.495047 10987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:42.495270 10987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:42.495293 10987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:42.496436 10987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:05:42.496476 10987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:42.496490 10987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:42.496502 10987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:05:42.496515 10987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:05:42.497928 10987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:38351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:36145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:36145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:42.498193 10987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:42.498446 10987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:42.503575 10993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:42.503578 10992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:42.503575 10995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:42.503618 10987 server_base.cc:1034] running on GCE node
I20260925 19:05:42.504004 10987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:42.504300 10987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:42.505465 10987 hybrid_clock.cc:648] HybridClock initialized: now 1790363142505451 us; error 41 us; skew 500 ppm
Sep 25 19:05:42 dist-test-slave-rn98 krb5kdc[10974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:42.507957 10987 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:05:42.508435 10987 webserver.cc:460] Webserver started at http://127.0.196.254:41259/ using document root <none> and password file <none>
I20260925 19:05:42.508649 10987 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:42.508710 10987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:42.508852 10987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:42.509897 10987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "3d66b43d9c8d44c0a95231c8ed1ddb61"
format_stamp: "Formatted at 2026-09-25 19:05:42 on dist-test-slave-rn98"
server_key: "7aa6e6efe57e25d4534780ae808fa711"
server_key_iv: "cde0a6e17dc5de77fa5d69a791ce18b1"
server_key_version: "encryptionkey@0"
I20260925 19:05:42.510254 10987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "3d66b43d9c8d44c0a95231c8ed1ddb61"
format_stamp: "Formatted at 2026-09-25 19:05:42 on dist-test-slave-rn98"
server_key: "7aa6e6efe57e25d4534780ae808fa711"
server_key_iv: "cde0a6e17dc5de77fa5d69a791ce18b1"
server_key_version: "encryptionkey@0"
I20260925 19:05:42.511624 10987 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:05:42.512339 11002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:42.512501 10987 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:05:42.512598 10987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "3d66b43d9c8d44c0a95231c8ed1ddb61"
format_stamp: "Formatted at 2026-09-25 19:05:42 on dist-test-slave-rn98"
server_key: "7aa6e6efe57e25d4534780ae808fa711"
server_key_iv: "cde0a6e17dc5de77fa5d69a791ce18b1"
server_key_version: "encryptionkey@0"
I20260925 19:05:42.512668 10987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:42.525605 10987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:42.527938 10987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:42.528127 10987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:42.532392 10987 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:36145
I20260925 19:05:42.532418 11054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:36145 every 8 connection(s)
I20260925 19:05:42.532783 10987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:05:42.533716 11055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:42.536159 11055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d66b43d9c8d44c0a95231c8ed1ddb61: Bootstrap starting.
I20260925 19:05:42.536785 11055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d66b43d9c8d44c0a95231c8ed1ddb61: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:42.537073 11055 log.cc:824] T 00000000000000000000000000000000 P 3d66b43d9c8d44c0a95231c8ed1ddb61: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:42.537832 11055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d66b43d9c8d44c0a95231c8ed1ddb61: No bootstrap required, opened a new log
I20260925 19:05:42.538825 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 10987
I20260925 19:05:42.538920 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:05:42.539098 787 external_mini_cluster.cc:1428] Setting key 508cccc5cf540ffe796daa84aaa58d3b
I20260925 19:05:42.539951 11055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d66b43d9c8d44c0a95231c8ed1ddb61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d66b43d9c8d44c0a95231c8ed1ddb61" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 36145 } }
I20260925 19:05:42.540094 11055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d66b43d9c8d44c0a95231c8ed1ddb61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:42.540117 11055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d66b43d9c8d44c0a95231c8ed1ddb61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d66b43d9c8d44c0a95231c8ed1ddb61, State: Initialized, Role: FOLLOWER
I20260925 19:05:42.540267 11055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d66b43d9c8d44c0a95231c8ed1ddb61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d66b43d9c8d44c0a95231c8ed1ddb61" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 36145 } }
I20260925 19:05:42.540364 11055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d66b43d9c8d44c0a95231c8ed1ddb61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:05:42.540408 11055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d66b43d9c8d44c0a95231c8ed1ddb61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:05:42.540477 11055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d66b43d9c8d44c0a95231c8ed1ddb61 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:42.541363 11055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d66b43d9c8d44c0a95231c8ed1ddb61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d66b43d9c8d44c0a95231c8ed1ddb61" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 36145 } }
I20260925 19:05:42.541482 11055 leader_election.cc:302] T 00000000000000000000000000000000 P 3d66b43d9c8d44c0a95231c8ed1ddb61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d66b43d9c8d44c0a95231c8ed1ddb61; no voters:
I20260925 19:05:42.541675 11055 leader_election.cc:288] T 00000000000000000000000000000000 P 3d66b43d9c8d44c0a95231c8ed1ddb61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:05:42.541694 11060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d66b43d9c8d44c0a95231c8ed1ddb61 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:05:42.541805 11060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d66b43d9c8d44c0a95231c8ed1ddb61 [term 1 LEADER]: Becoming Leader. State: Replica: 3d66b43d9c8d44c0a95231c8ed1ddb61, State: Running, Role: LEADER
I20260925 19:05:42.541896 11060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d66b43d9c8d44c0a95231c8ed1ddb61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d66b43d9c8d44c0a95231c8ed1ddb61" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 36145 } }
I20260925 19:05:42.541939 11055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d66b43d9c8d44c0a95231c8ed1ddb61 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:05:42.542208 11060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d66b43d9c8d44c0a95231c8ed1ddb61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d66b43d9c8d44c0a95231c8ed1ddb61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d66b43d9c8d44c0a95231c8ed1ddb61" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 36145 } } }
I20260925 19:05:42.542279 11060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d66b43d9c8d44c0a95231c8ed1ddb61 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:42.542474 11060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d66b43d9c8d44c0a95231c8ed1ddb61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d66b43d9c8d44c0a95231c8ed1ddb61. Latest consensus state: current_term: 1 leader_uuid: "3d66b43d9c8d44c0a95231c8ed1ddb61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d66b43d9c8d44c0a95231c8ed1ddb61" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 36145 } } }
I20260925 19:05:42.542549 11066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:05:42.543207 11060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d66b43d9c8d44c0a95231c8ed1ddb61 [sys.catalog]: This master's current role is: LEADER
I20260925 19:05:42.543236 11066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:05:42.545678 11066 catalog_manager.cc:1347] Generated new cluster ID: 55755fe3deff4cc78e85a3f05e65983d
I20260925 19:05:42.545734 11066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:05:42 dist-test-slave-rn98 krb5kdc[10974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:05:42.550853 11059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:42.539666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53088 (local address 127.0.196.254:36145)
0925 19:05:42.539883 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:42.539895 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:42.539934 (+ 39us) server_negotiation.cc:407] Connection header received
0925 19:05:42.540045 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:42.540050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:42.540110 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:42.540262 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:42.540964 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:42.542112 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:42.543178 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:42.543775 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:42.543906 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:42.546820 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:42.546824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:42.546824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:42.546836 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:42.548910 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:42.549288 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:42.549291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:42.549292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:42.549403 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:42.549566 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:42.549567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:42.549567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:42.549859 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:42.549998 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:42.550468 (+ 470us) server_negotiation.cc:299] Negotiation successful
0925 19:05:42.550605 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":97,"threads_started":1}
I20260925 19:05:42.554078 11066 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:05:42.554857 11066 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:05:42.566162 11066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d66b43d9c8d44c0a95231c8ed1ddb61: Generated new TSK 0
I20260925 19:05:42.566399 11066 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
I20260925 19:05:42.598665 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:0
--local_ip_for_outbound_sockets=127.0.196.193
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:36145
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:38351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:42.682570 11081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:42.682843 11081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:42.682881 11081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:42.684051 11081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:42.684118 11081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:42.684162 11081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:05:42.685644 11081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:38351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:36145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:42.685920 11081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:42.686149 11081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:05:42.691318 11081 server_base.cc:1034] running on GCE node
W20260925 19:05:42.691299 11087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:42.691293 11089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:42.691295 11086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:42.691826 11081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:42.692029 11081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:42.693169 11081 hybrid_clock.cc:648] HybridClock initialized: now 1790363142693158 us; error 28 us; skew 500 ppm
Sep 25 19:05:42 dist-test-slave-rn98 krb5kdc[10974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:42.695844 11081 init.cc:377] Logged in from keytab as kudu/127.0.196.193@KRBTEST.COM (short username kudu)
I20260925 19:05:42.696292 11081 webserver.cc:460] Webserver started at http://127.0.196.193:34139/ using document root <none> and password file <none>
I20260925 19:05:42.696538 11081 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:42.696607 11081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:42.696749 11081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:42.697808 11081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-0/data/instance:
uuid: "c48d16162ccb4790a9c0761f4775cc8a"
format_stamp: "Formatted at 2026-09-25 19:05:42 on dist-test-slave-rn98"
server_key: "d14cb86905d92ca505ed0904b610ca91"
server_key_iv: "7ea84a866c9ae89fda7de2a7e5ef974d"
server_key_version: "encryptionkey@0"
I20260925 19:05:42.698164 11081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-0/wal/instance:
uuid: "c48d16162ccb4790a9c0761f4775cc8a"
format_stamp: "Formatted at 2026-09-25 19:05:42 on dist-test-slave-rn98"
server_key: "d14cb86905d92ca505ed0904b610ca91"
server_key_iv: "7ea84a866c9ae89fda7de2a7e5ef974d"
server_key_version: "encryptionkey@0"
I20260925 19:05:42.699558 11081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:05:42.700409 11096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:42.700587 11081 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:42.700686 11081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "c48d16162ccb4790a9c0761f4775cc8a"
format_stamp: "Formatted at 2026-09-25 19:05:42 on dist-test-slave-rn98"
server_key: "d14cb86905d92ca505ed0904b610ca91"
server_key_iv: "7ea84a866c9ae89fda7de2a7e5ef974d"
server_key_version: "encryptionkey@0"
I20260925 19:05:42.700762 11081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:42.710086 11081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:42.712445 11081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:42.712620 11081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:42.712880 11081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:42.713225 11081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:42.713289 11081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:42.713335 11081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:42.713371 11081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:42.719895 11081 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:41175
I20260925 19:05:42.719940 11209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:41175 every 8 connection(s)
I20260925 19:05:42.720290 11081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
I20260925 19:05:42.723852 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 11081
I20260925 19:05:42.723945 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-0/wal/instance
I20260925 19:05:42.724114 787 external_mini_cluster.cc:1428] Setting key fb6692432ff3068f2fc7232e9c3ae0bb
Sep 25 19:05:42 dist-test-slave-rn98 krb5kdc[10974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:05:42.731148 11059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:42.721116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:47571 (local address 127.0.196.254:36145)
0925 19:05:42.721180 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:42.721182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:42.721390 (+ 208us) server_negotiation.cc:407] Connection header received
0925 19:05:42.721962 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:42.721963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:42.721975 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:42.722029 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:42.723052 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:42.723569 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:42.724961 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:42.725180 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:42.725283 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:42.727948 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:42.727950 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:42.727950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:42.727958 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:42.729590 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:42.730096 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:42.730096 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:42.730097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:42.730153 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:42.730433 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:42.730433 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:42.730433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:42.730544 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:42.730598 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:42.730999 (+ 401us) server_negotiation.cc:299] Negotiation successful
0925 19:05:42.731081 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 19:05:42.731389 11212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:42.721092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:36145 (local address 127.0.196.193:47571)
0925 19:05:42.721265 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:42.721302 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:42.721853 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:42.722112 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:42.722116 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:42.722762 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:42.722956 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:42.722961 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:42.723658 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:42.723659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:42.724837 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:42.724840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:42.725291 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:42.725292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:42.725421 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:42.725946 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:42.725968 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:42.727846 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:42.729679 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:42.729687 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:42.729698 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:42.730013 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:42.730222 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:42.730223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:42.730223 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:42.730366 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:42.730609 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:42.730645 (+ 36us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:42.730827 (+ 182us) client_negotiation.cc:769] Sending connection context
0925 19:05:42.730958 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 19:05:42.731130 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
I20260925 19:05:42.731990 11210 heartbeater.cc:347] Connected to a master server at 127.0.196.254:36145
I20260925 19:05:42.732098 11210 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:42.732301 11210 heartbeater.cc:511] Master 127.0.196.254:36145 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.196.194@KRBTEST.COM; defaulting to no policy
I20260925 19:05:42.732885 11019 ts_manager.cc:194] Registered new tserver with Master: c48d16162ccb4790a9c0761f4775cc8a (127.0.196.193:41175)
I20260925 19:05:42.733484 11019 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.193@KRBTEST.COM'} at 127.0.196.193:47571
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
I20260925 19:05:42.751959 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:0
--local_ip_for_outbound_sockets=127.0.196.194
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:36145
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:38351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:42.832051 11215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:42.832270 11215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:42.832295 11215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:42.833503 11215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:42.833551 11215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:42.833578 11215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:05:42.835100 11215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:38351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:36145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:42.835358 11215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:42.835601 11215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:42.840610 11221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:42.840617 11223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:42.840642 11215 server_base.cc:1034] running on GCE node
W20260925 19:05:42.840626 11220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:42.841094 11215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:42.841305 11215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:42.842465 11215 hybrid_clock.cc:648] HybridClock initialized: now 1790363142842453 us; error 39 us; skew 500 ppm
Sep 25 19:05:42 dist-test-slave-rn98 krb5kdc[10974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:42.845006 11215 init.cc:377] Logged in from keytab as kudu/127.0.196.194@KRBTEST.COM (short username kudu)
I20260925 19:05:42.845468 11215 webserver.cc:460] Webserver started at http://127.0.196.194:46775/ using document root <none> and password file <none>
I20260925 19:05:42.845687 11215 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:42.845749 11215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:42.845892 11215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:42.847048 11215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-1/data/instance:
uuid: "8d77eaa3f6af468187a99e3dbbf4954e"
format_stamp: "Formatted at 2026-09-25 19:05:42 on dist-test-slave-rn98"
server_key: "ba6bc13475bffb23f3e7628f061fb6ee"
server_key_iv: "4a7f65cd684d91622d6bf2763409fe4f"
server_key_version: "encryptionkey@0"
I20260925 19:05:42.847410 11215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-1/wal/instance:
uuid: "8d77eaa3f6af468187a99e3dbbf4954e"
format_stamp: "Formatted at 2026-09-25 19:05:42 on dist-test-slave-rn98"
server_key: "ba6bc13475bffb23f3e7628f061fb6ee"
server_key_iv: "4a7f65cd684d91622d6bf2763409fe4f"
server_key_version: "encryptionkey@0"
I20260925 19:05:42.848748 11215 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:42.849484 11230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:42.849695 11215 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:42.849793 11215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "8d77eaa3f6af468187a99e3dbbf4954e"
format_stamp: "Formatted at 2026-09-25 19:05:42 on dist-test-slave-rn98"
server_key: "ba6bc13475bffb23f3e7628f061fb6ee"
server_key_iv: "4a7f65cd684d91622d6bf2763409fe4f"
server_key_version: "encryptionkey@0"
I20260925 19:05:42.849864 11215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:42.870249 11215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:42.872618 11215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:42.872880 11215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:42.873131 11215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:42.873451 11215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:42.873500 11215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:42.873544 11215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:42.873574 11215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:42.879377 11215 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:42309
I20260925 19:05:42.879508 11343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:42309 every 8 connection(s)
I20260925 19:05:42.879806 11215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:05:42 dist-test-slave-rn98 krb5kdc[10974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:05:42.887179 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 11215
I20260925 19:05:42.887277 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-1/wal/instance
I20260925 19:05:42.887503 787 external_mini_cluster.cc:1428] Setting key 9041eb1e5f95d109d9cd48a52c359cc4
I20260925 19:05:42.890435 11346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:42.880559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:36145 (local address 127.0.196.194:41677)
0925 19:05:42.880704 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:42.880757 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:42.881510 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:42.881816 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:42.881820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:42.882465 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:42.882741 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:42.882746 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:42.883417 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:42.883418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:42.884246 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:42.884248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:42.884643 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:42.884644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:42.884764 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:42.885304 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:42.885313 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:42.886767 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:42.888610 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:42.888619 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:42.888622 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:42.889028 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:42.889253 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:42.889254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:42.889255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:42.889413 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:42.889847 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:42.889855 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:42.890029 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 19:05:42.890122 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 19:05:42.890258 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
I20260925 19:05:42.890439 11059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:42.880587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:41677 (local address 127.0.196.254:36145)
0925 19:05:42.880635 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:42.880638 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:42.880869 (+ 231us) server_negotiation.cc:407] Connection header received
0925 19:05:42.881648 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:42.881649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:42.881661 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:42.881729 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:42.882843 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:42.883337 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:42.884352 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:42.884561 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:42.884634 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:42.886864 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:42.886866 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:42.886866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:42.886873 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:42.888489 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:42.889118 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:42.889120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:42.889120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:42.889190 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:42.889504 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:42.889505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:42.889506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:42.889640 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:42.889845 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:42.890191 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 19:05:42.890317 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:05:42.891043 11344 heartbeater.cc:347] Connected to a master server at 127.0.196.254:36145
I20260925 19:05:42.891155 11344 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:42.891340 11344 heartbeater.cc:511] Master 127.0.196.254:36145 requested a full tablet report, sending...
I20260925 19:05:42.891726 11019 ts_manager.cc:194] Registered new tserver with Master: 8d77eaa3f6af468187a99e3dbbf4954e (127.0.196.194:42309)
I20260925 19:05:42.892278 11019 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.194@KRBTEST.COM'} at 127.0.196.194:41677
WARNING: no policy specified for kudu/127.0.196.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
I20260925 19:05:42.915498 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:0
--local_ip_for_outbound_sockets=127.0.196.195
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:36145
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:38351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:05:42.996794 11349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:05:42.997044 11349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:05:42.997082 11349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:05:42.998253 11349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:05:42.998324 11349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:05:42.998368 11349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:05:42.999864 11349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:38351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:36145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:05:43.000128 11349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:05:43.000355 11349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:05:43.005592 11355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:43.005761 11349 server_base.cc:1034] running on GCE node
W20260925 19:05:43.005584 11354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:05:43.005585 11357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:05:43.006073 11349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:05:43.006301 11349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:05:43.007521 11349 hybrid_clock.cc:648] HybridClock initialized: now 1790363143007509 us; error 33 us; skew 500 ppm
Sep 25 19:05:43 dist-test-slave-rn98 krb5kdc[10974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:05:43.010275 11349 init.cc:377] Logged in from keytab as kudu/127.0.196.195@KRBTEST.COM (short username kudu)
I20260925 19:05:43.010793 11349 webserver.cc:460] Webserver started at http://127.0.196.195:44513/ using document root <none> and password file <none>
I20260925 19:05:43.011030 11349 fs_manager.cc:359] Metadata directory not provided
I20260925 19:05:43.011076 11349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:05:43.011209 11349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:05:43.012341 11349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-2/data/instance:
uuid: "69f01d88414e48719504a5395c560bc0"
format_stamp: "Formatted at 2026-09-25 19:05:43 on dist-test-slave-rn98"
server_key: "9b16eb3999fea753919544007e8ef8e3"
server_key_iv: "f27b3015ed01a6f69768cc2c12899fc4"
server_key_version: "encryptionkey@0"
I20260925 19:05:43.012737 11349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-2/wal/instance:
uuid: "69f01d88414e48719504a5395c560bc0"
format_stamp: "Formatted at 2026-09-25 19:05:43 on dist-test-slave-rn98"
server_key: "9b16eb3999fea753919544007e8ef8e3"
server_key_iv: "f27b3015ed01a6f69768cc2c12899fc4"
server_key_version: "encryptionkey@0"
I20260925 19:05:43.014102 11349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:05:43.014920 11364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:43.015154 11349 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:05:43.015237 11349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "69f01d88414e48719504a5395c560bc0"
format_stamp: "Formatted at 2026-09-25 19:05:43 on dist-test-slave-rn98"
server_key: "9b16eb3999fea753919544007e8ef8e3"
server_key_iv: "f27b3015ed01a6f69768cc2c12899fc4"
server_key_version: "encryptionkey@0"
I20260925 19:05:43.015293 11349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:05:43.052649 11349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:05:43.054708 11349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:05:43.054822 11349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:05:43.055013 11349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:05:43.055333 11349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:05:43.055367 11349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:43.055419 11349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:05:43.055450 11349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:05:43.061178 11349 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:43197
I20260925 19:05:43.061249 11477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:43197 every 8 connection(s)
I20260925 19:05:43.061563 11349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
Sep 25 19:05:43 dist-test-slave-rn98 krb5kdc[10974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:05:43.070847 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 11349
I20260925 19:05:43.070945 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-2/wal/instance
I20260925 19:05:43.071019 11059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:43.062279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:48357 (local address 127.0.196.254:36145)
0925 19:05:43.062346 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:43.062348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:43.062546 (+ 198us) server_negotiation.cc:407] Connection header received
0925 19:05:43.063150 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:43.063151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:43.063161 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:43.063227 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:43.064170 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:43.064663 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:43.065696 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:43.065893 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:43.065948 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:43.068213 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:43.068214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:43.068214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:43.068221 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:43.069617 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:43.069968 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:43.069968 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:43.069969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:43.070018 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:43.070227 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:43.070228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:43.070228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:43.070345 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:43.070397 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:43.070815 (+ 418us) server_negotiation.cc:299] Negotiation successful
0925 19:05:43.070906 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 19:05:43.071134 787 external_mini_cluster.cc:1428] Setting key b13cc113b3d48d79bbbf6e2a54a4d2c9
I20260925 19:05:43.071245 11480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:43.062276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:36145 (local address 127.0.196.195:48357)
0925 19:05:43.062421 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:43.062458 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:43.062982 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:43.063308 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:43.063311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:43.063880 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:05:43.064078 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:43.064083 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:43.064774 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:43.064775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:43.065615 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:43.065618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:43.065991 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:43.065992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:43.066142 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:43.066766 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:05:43.066775 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:05:43.068130 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:05:43.069690 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:43.069696 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:43.069698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:43.069906 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:43.070073 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:05:43.070073 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:05:43.070074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:05:43.070176 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:05:43.070419 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:05:43.070425 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:05:43.070553 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 19:05:43.070780 (+ 227us) client_negotiation.cc:240] Negotiation successful
0925 19:05:43.070945 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260925 19:05:43.071935 11478 heartbeater.cc:347] Connected to a master server at 127.0.196.254:36145
I20260925 19:05:43.072047 11478 heartbeater.cc:464] Registering TS with master...
I20260925 19:05:43.072237 11478 heartbeater.cc:511] Master 127.0.196.254:36145 requested a full tablet report, sending...
I20260925 19:05:43.072595 11019 ts_manager.cc:194] Registered new tserver with Master: 69f01d88414e48719504a5395c560bc0 (127.0.196.195:43197)
I20260925 19:05:43.073012 11019 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.195@KRBTEST.COM'} at 127.0.196.195:48357
I20260925 19:05:43.083236 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:05:43.090783 11059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:43.084646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53098 (local address 127.0.196.254:36145)
0925 19:05:43.084737 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:43.084739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:43.084848 (+ 109us) server_negotiation.cc:407] Connection header received
0925 19:05:43.084926 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:43.084928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:43.084938 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:43.084979 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:05:43.085570 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:43.086074 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:43.086884 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:43.087085 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:43.087174 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:43.088417 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:05:43.088419 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:05:43.088419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:05:43.088428 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:05:43.089595 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:43.089932 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:43.089933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:43.089934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:43.089995 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:05:43.090194 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:05:43.090195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:05:43.090195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:05:43.090346 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:05:43.090450 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:43.090572 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 19:05:43.090678 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 19:05:43.092310 11019 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:05:43.093901 11019 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:05:43.100396 11491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:43.096282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:41175 (local address 127.0.0.1:56098)
0925 19:05:43.096679 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:43.096690 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:43.096730 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:43.097077 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:43.097079 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:43.097081 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:43.097166 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:43.097168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:43.098071 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:43.098072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:43.099316 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:43.099318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:43.100098 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:43.100099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:43.100174 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:43.100188 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 19:05:43.100256 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 19:05:43.100307 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":5,"thread_start_us":55,"threads_started":1}
I20260925 19:05:43.100399 11494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:43.096669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56098 (local address 127.0.196.193:41175)
0925 19:05:43.096863 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:43.096866 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:43.096875 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:05:43.096911 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:43.096913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:43.096926 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:43.096997 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:43.097248 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:43.097974 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:43.099402 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:43.100012 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:43.100099 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:43.100160 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:43.100269 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 19:05:43.100307 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":103,"threads_started":1}
I20260925 19:05:43.100853 11489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:43.096299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:43197 (local address 127.0.0.1:57888)
0925 19:05:43.096845 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:43.096854 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:43.096952 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:43.097221 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:43.097222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:43.097223 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:43.097253 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:43.097254 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:43.098501 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:43.098502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:43.099517 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:43.099519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:43.100633 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:43.100634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:43.100691 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:43.100695 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:05:43.100741 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 19:05:43.100778 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260925 19:05:43.100896 11492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:43.096463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57888 (local address 127.0.196.195:43197)
0925 19:05:43.096643 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:43.096649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:43.096926 (+ 277us) server_negotiation.cc:407] Connection header received
0925 19:05:43.097055 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:43.097057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:43.097073 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:43.097147 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:43.097346 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:43.098402 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:43.099621 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:43.100477 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:43.100545 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:43.100600 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:43.100773 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 19:05:43.100820 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":99,"threads_started":1}
I20260925 19:05:43.101131 11144 tablet_service.cc:1467] Processing CreateTablet for tablet 3466ed3f6d94412e941cc0caa78f227e (DEFAULT_TABLE table=test-table [id=d1bd05724692487ab95c193592d99f74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:05:43.101387 11493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:43.096532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46284 (local address 127.0.196.194:42309)
0925 19:05:43.096709 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:43.096712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:43.096722 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:05:43.096808 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:43.096811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:43.096828 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:43.096921 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:43.097172 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:43.098349 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:43.099604 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:43.100346 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:43.100419 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:43.101159 (+ 740us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:43.101259 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 19:05:43.101302 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":80,"threads_started":1}
I20260925 19:05:43.101404 11490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:43.096282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:42309 (local address 127.0.0.1:46284)
0925 19:05:43.096463 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:43.096508 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:43.096577 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:43.096936 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:43.096940 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:43.096942 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:43.097087 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:43.097090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:43.098454 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:43.098455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:43.099505 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:43.099507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:43.100824 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:43.100825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:43.100883 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:43.100887 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:05:43.101244 (+ 357us) client_negotiation.cc:240] Negotiation successful
0925 19:05:43.101335 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":2,"thread_start_us":93,"threads_started":1}
I20260925 19:05:43.101507 11144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3466ed3f6d94412e941cc0caa78f227e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:43.102330 11278 tablet_service.cc:1467] Processing CreateTablet for tablet 3466ed3f6d94412e941cc0caa78f227e (DEFAULT_TABLE table=test-table [id=d1bd05724692487ab95c193592d99f74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:05:43.102689 11278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3466ed3f6d94412e941cc0caa78f227e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:43.103241 11412 tablet_service.cc:1467] Processing CreateTablet for tablet 3466ed3f6d94412e941cc0caa78f227e (DEFAULT_TABLE table=test-table [id=d1bd05724692487ab95c193592d99f74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:05:43.103579 11412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3466ed3f6d94412e941cc0caa78f227e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:05:43.105079 11495 tablet_bootstrap.cc:492] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a: Bootstrap starting.
I20260925 19:05:43.105746 11495 tablet_bootstrap.cc:654] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:43.105937 11496 tablet_bootstrap.cc:492] T 3466ed3f6d94412e941cc0caa78f227e P 8d77eaa3f6af468187a99e3dbbf4954e: Bootstrap starting.
I20260925 19:05:43.106328 11495 log.cc:824] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:43.106488 11496 tablet_bootstrap.cc:654] T 3466ed3f6d94412e941cc0caa78f227e P 8d77eaa3f6af468187a99e3dbbf4954e: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:43.106755 11498 tablet_bootstrap.cc:492] T 3466ed3f6d94412e941cc0caa78f227e P 69f01d88414e48719504a5395c560bc0: Bootstrap starting.
I20260925 19:05:43.107095 11496 log.cc:824] T 3466ed3f6d94412e941cc0caa78f227e P 8d77eaa3f6af468187a99e3dbbf4954e: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:43.107177 11495 tablet_bootstrap.cc:492] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a: No bootstrap required, opened a new log
I20260925 19:05:43.107251 11495 ts_tablet_manager.cc:1397] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:05:43.107462 11498 tablet_bootstrap.cc:654] T 3466ed3f6d94412e941cc0caa78f227e P 69f01d88414e48719504a5395c560bc0: Neither blocks nor log segments found. Creating new log.
I20260925 19:05:43.107950 11496 tablet_bootstrap.cc:492] T 3466ed3f6d94412e941cc0caa78f227e P 8d77eaa3f6af468187a99e3dbbf4954e: No bootstrap required, opened a new log
I20260925 19:05:43.108062 11498 log.cc:824] T 3466ed3f6d94412e941cc0caa78f227e P 69f01d88414e48719504a5395c560bc0: Log is configured to *not* fsync() on all Append() calls
I20260925 19:05:43.108081 11496 ts_tablet_manager.cc:1397] T 3466ed3f6d94412e941cc0caa78f227e P 8d77eaa3f6af468187a99e3dbbf4954e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:05:43.108836 11498 tablet_bootstrap.cc:492] T 3466ed3f6d94412e941cc0caa78f227e P 69f01d88414e48719504a5395c560bc0: No bootstrap required, opened a new log
I20260925 19:05:43.108908 11498 ts_tablet_manager.cc:1397] T 3466ed3f6d94412e941cc0caa78f227e P 69f01d88414e48719504a5395c560bc0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:05:43.109093 11495 raft_consensus.cc:348] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c48d16162ccb4790a9c0761f4775cc8a" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 41175 } } peers { permanent_uuid: "8d77eaa3f6af468187a99e3dbbf4954e" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42309 } } peers { permanent_uuid: "69f01d88414e48719504a5395c560bc0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43197 } }
I20260925 19:05:43.109269 11495 raft_consensus.cc:374] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:43.109311 11495 raft_consensus.cc:729] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c48d16162ccb4790a9c0761f4775cc8a, State: Initialized, Role: FOLLOWER
I20260925 19:05:43.109428 11495 consensus_queue.cc:260] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c48d16162ccb4790a9c0761f4775cc8a" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 41175 } } peers { permanent_uuid: "8d77eaa3f6af468187a99e3dbbf4954e" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42309 } } peers { permanent_uuid: "69f01d88414e48719504a5395c560bc0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43197 } }
I20260925 19:05:43.109689 11495 ts_tablet_manager.cc:1428] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 19:05:43.109833 11210 heartbeater.cc:503] Master 127.0.196.254:36145 was elected leader, sending a full tablet report...
I20260925 19:05:43.110414 11496 raft_consensus.cc:348] T 3466ed3f6d94412e941cc0caa78f227e P 8d77eaa3f6af468187a99e3dbbf4954e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c48d16162ccb4790a9c0761f4775cc8a" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 41175 } } peers { permanent_uuid: "8d77eaa3f6af468187a99e3dbbf4954e" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42309 } } peers { permanent_uuid: "69f01d88414e48719504a5395c560bc0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43197 } }
I20260925 19:05:43.110602 11496 raft_consensus.cc:374] T 3466ed3f6d94412e941cc0caa78f227e P 8d77eaa3f6af468187a99e3dbbf4954e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:43.110697 11496 raft_consensus.cc:729] T 3466ed3f6d94412e941cc0caa78f227e P 8d77eaa3f6af468187a99e3dbbf4954e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d77eaa3f6af468187a99e3dbbf4954e, State: Initialized, Role: FOLLOWER
I20260925 19:05:43.110816 11496 consensus_queue.cc:260] T 3466ed3f6d94412e941cc0caa78f227e P 8d77eaa3f6af468187a99e3dbbf4954e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c48d16162ccb4790a9c0761f4775cc8a" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 41175 } } peers { permanent_uuid: "8d77eaa3f6af468187a99e3dbbf4954e" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42309 } } peers { permanent_uuid: "69f01d88414e48719504a5395c560bc0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43197 } }
I20260925 19:05:43.110984 11498 raft_consensus.cc:348] T 3466ed3f6d94412e941cc0caa78f227e P 69f01d88414e48719504a5395c560bc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c48d16162ccb4790a9c0761f4775cc8a" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 41175 } } peers { permanent_uuid: "8d77eaa3f6af468187a99e3dbbf4954e" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42309 } } peers { permanent_uuid: "69f01d88414e48719504a5395c560bc0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43197 } }
I20260925 19:05:43.111100 11496 ts_tablet_manager.cc:1428] T 3466ed3f6d94412e941cc0caa78f227e P 8d77eaa3f6af468187a99e3dbbf4954e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 19:05:43.111114 11344 heartbeater.cc:503] Master 127.0.196.254:36145 was elected leader, sending a full tablet report...
I20260925 19:05:43.111155 11498 raft_consensus.cc:374] T 3466ed3f6d94412e941cc0caa78f227e P 69f01d88414e48719504a5395c560bc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:05:43.111191 11498 raft_consensus.cc:729] T 3466ed3f6d94412e941cc0caa78f227e P 69f01d88414e48719504a5395c560bc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f01d88414e48719504a5395c560bc0, State: Initialized, Role: FOLLOWER
I20260925 19:05:43.111302 11498 consensus_queue.cc:260] T 3466ed3f6d94412e941cc0caa78f227e P 69f01d88414e48719504a5395c560bc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c48d16162ccb4790a9c0761f4775cc8a" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 41175 } } peers { permanent_uuid: "8d77eaa3f6af468187a99e3dbbf4954e" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42309 } } peers { permanent_uuid: "69f01d88414e48719504a5395c560bc0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43197 } }
I20260925 19:05:43.111558 11498 ts_tablet_manager.cc:1428] T 3466ed3f6d94412e941cc0caa78f227e P 69f01d88414e48719504a5395c560bc0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:05:43.111765 11478 heartbeater.cc:503] Master 127.0.196.254:36145 was elected leader, sending a full tablet report...
I20260925 19:05:43.124788 11501 raft_consensus.cc:482] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:05:43.124931 11501 raft_consensus.cc:504] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c48d16162ccb4790a9c0761f4775cc8a" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 41175 } } peers { permanent_uuid: "8d77eaa3f6af468187a99e3dbbf4954e" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42309 } } peers { permanent_uuid: "69f01d88414e48719504a5395c560bc0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43197 } }
I20260925 19:05:43.125300 11501 leader_election.cc:288] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d77eaa3f6af468187a99e3dbbf4954e (127.0.196.194:42309), 69f01d88414e48719504a5395c560bc0 (127.0.196.195:43197)
I20260925 19:05:43.128178 11493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:43.125548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:47941 (local address 127.0.196.194:42309)
0925 19:05:43.125595 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:43.125597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:43.125639 (+ 42us) server_negotiation.cc:407] Connection header received
0925 19:05:43.125679 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:43.125680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:43.125687 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:43.125746 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:43.125909 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:43.126392 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:43.127436 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:43.127816 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:43.127910 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:43.127920 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:43.128069 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 19:05:43.128106 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:05:43.128189 11212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:43.125495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:42309 (local address 127.0.196.193:47941)
0925 19:05:43.125537 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:43.125549 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:43.125590 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:43.125769 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:43.125771 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:43.125773 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:43.125844 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:43.125845 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:43.126484 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:43.126485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:43.127341 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:43.127344 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:43.127910 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:43.127911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:43.127997 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:43.128001 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:05:43.128065 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 19:05:43.128104 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 19:05:43.128203 11492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:43.125570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:52651 (local address 127.0.196.195:43197)
0925 19:05:43.125622 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:43.125624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:43.125785 (+ 161us) server_negotiation.cc:407] Connection header received
0925 19:05:43.125846 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:43.125847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:43.125857 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:43.125926 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:05:43.126064 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:43.126524 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:43.127473 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:43.127856 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:43.127947 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:43.127956 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:43.128099 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 19:05:43.128137 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 19:05:43.128262 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:43.125495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:43197 (local address 127.0.196.193:52651)
0925 19:05:43.125697 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:43.125705 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 19:05:43.125792 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:05:43.125927 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:05:43.125928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:05:43.125929 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:05:43.126000 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:43.126001 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:43.126611 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:43.126612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:43.127384 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:05:43.127386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:05:43.127948 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:05:43.127949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:05:43.128020 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:43.128023 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:05:43.128091 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 19:05:43.128130 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":50,"thread_start_us":107,"threads_started":1}
I20260925 19:05:43.128566 11298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3466ed3f6d94412e941cc0caa78f227e" candidate_uuid: "c48d16162ccb4790a9c0761f4775cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d77eaa3f6af468187a99e3dbbf4954e" is_pre_election: true
I20260925 19:05:43.128578 11432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3466ed3f6d94412e941cc0caa78f227e" candidate_uuid: "c48d16162ccb4790a9c0761f4775cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f01d88414e48719504a5395c560bc0" is_pre_election: true
I20260925 19:05:43.128731 11432 raft_consensus.cc:2454] T 3466ed3f6d94412e941cc0caa78f227e P 69f01d88414e48719504a5395c560bc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c48d16162ccb4790a9c0761f4775cc8a in term 0.
I20260925 19:05:43.128731 11298 raft_consensus.cc:2454] T 3466ed3f6d94412e941cc0caa78f227e P 8d77eaa3f6af468187a99e3dbbf4954e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c48d16162ccb4790a9c0761f4775cc8a in term 0.
I20260925 19:05:43.128983 11097 leader_election.cc:302] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d77eaa3f6af468187a99e3dbbf4954e, c48d16162ccb4790a9c0761f4775cc8a; no voters:
I20260925 19:05:43.129170 11501 raft_consensus.cc:2781] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:05:43.129211 11501 raft_consensus.cc:482] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:05:43.129227 11501 raft_consensus.cc:3037] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:43.129889 11501 raft_consensus.cc:504] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c48d16162ccb4790a9c0761f4775cc8a" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 41175 } } peers { permanent_uuid: "8d77eaa3f6af468187a99e3dbbf4954e" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42309 } } peers { permanent_uuid: "69f01d88414e48719504a5395c560bc0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43197 } }
I20260925 19:05:43.130044 11501 leader_election.cc:288] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a [CANDIDATE]: Term 1 election: Requested vote from peers 8d77eaa3f6af468187a99e3dbbf4954e (127.0.196.194:42309), 69f01d88414e48719504a5395c560bc0 (127.0.196.195:43197)
I20260925 19:05:43.130172 11298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3466ed3f6d94412e941cc0caa78f227e" candidate_uuid: "c48d16162ccb4790a9c0761f4775cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d77eaa3f6af468187a99e3dbbf4954e"
I20260925 19:05:43.130215 11432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3466ed3f6d94412e941cc0caa78f227e" candidate_uuid: "c48d16162ccb4790a9c0761f4775cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f01d88414e48719504a5395c560bc0"
I20260925 19:05:43.130250 11298 raft_consensus.cc:3037] T 3466ed3f6d94412e941cc0caa78f227e P 8d77eaa3f6af468187a99e3dbbf4954e [term 0 FOLLOWER]: Advancing to term 1
W20260925 19:05:43.130337 11345 tablet.cc:2369] T 3466ed3f6d94412e941cc0caa78f227e P 8d77eaa3f6af468187a99e3dbbf4954e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:05:43.130317 11432 raft_consensus.cc:3037] T 3466ed3f6d94412e941cc0caa78f227e P 69f01d88414e48719504a5395c560bc0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:05:43.131069 11298 raft_consensus.cc:2454] T 3466ed3f6d94412e941cc0caa78f227e P 8d77eaa3f6af468187a99e3dbbf4954e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c48d16162ccb4790a9c0761f4775cc8a in term 1.
I20260925 19:05:43.131153 11432 raft_consensus.cc:2454] T 3466ed3f6d94412e941cc0caa78f227e P 69f01d88414e48719504a5395c560bc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c48d16162ccb4790a9c0761f4775cc8a in term 1.
I20260925 19:05:43.131234 11097 leader_election.cc:302] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d77eaa3f6af468187a99e3dbbf4954e, c48d16162ccb4790a9c0761f4775cc8a; no voters:
I20260925 19:05:43.131340 11501 raft_consensus.cc:2781] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:05:43.131529 11501 raft_consensus.cc:686] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a [term 1 LEADER]: Becoming Leader. State: Replica: c48d16162ccb4790a9c0761f4775cc8a, State: Running, Role: LEADER
I20260925 19:05:43.131635 11501 consensus_queue.cc:237] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c48d16162ccb4790a9c0761f4775cc8a" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 41175 } } peers { permanent_uuid: "8d77eaa3f6af468187a99e3dbbf4954e" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42309 } } peers { permanent_uuid: "69f01d88414e48719504a5395c560bc0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43197 } }
I20260925 19:05:43.132543 11018 catalog_manager.cc:5495] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a reported cstate change: term changed from 0 to 1, leader changed from <none> to c48d16162ccb4790a9c0761f4775cc8a (127.0.196.193). New cstate: current_term: 1 leader_uuid: "c48d16162ccb4790a9c0761f4775cc8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c48d16162ccb4790a9c0761f4775cc8a" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 41175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d77eaa3f6af468187a99e3dbbf4954e" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f01d88414e48719504a5395c560bc0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43197 } health_report { overall_health: UNKNOWN } } }
I20260925 19:05:43.146181 11494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:43.143896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56102 (local address 127.0.196.193:41175)
0925 19:05:43.143937 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:05:43.143938 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:05:43.143950 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:05:43.144024 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:05:43.144025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:05:43.144042 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:05:43.144103 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:05:43.144210 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:43.144660 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:43.145403 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:05:43.145578 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:05:43.145631 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:05:43.145803 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:05:43.145901 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:05:43.145960 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 19:05:43.146090 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 19:05:43.146106 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:05:43.148032 11298 raft_consensus.cc:1261] T 3466ed3f6d94412e941cc0caa78f227e P 8d77eaa3f6af468187a99e3dbbf4954e [term 1 FOLLOWER]: Refusing update from remote peer c48d16162ccb4790a9c0761f4775cc8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:05:43.148032 11432 raft_consensus.cc:1261] T 3466ed3f6d94412e941cc0caa78f227e P 69f01d88414e48719504a5395c560bc0 [term 1 FOLLOWER]: Refusing update from remote peer c48d16162ccb4790a9c0761f4775cc8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:05:43.148430 11505 consensus_queue.cc:1035] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f01d88414e48719504a5395c560bc0" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:05:43.148567 11501 consensus_queue.cc:1035] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d77eaa3f6af468187a99e3dbbf4954e" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:05:43.150933 11508 mvcc.cc:205] Tried to move back new op lower bound from 7333327434331398144 to 7333327434267914240. Current Snapshot: MvccSnapshot[applied={T|T < 7333327434331398144}]
I20260925 19:05:43.151127 11509 mvcc.cc:205] Tried to move back new op lower bound from 7333327434331398144 to 7333327434267914240. Current Snapshot: MvccSnapshot[applied={T|T < 7333327434331398144}]
I20260925 19:05:43.151340 11510 mvcc.cc:205] Tried to move back new op lower bound from 7333327434331398144 to 7333327434267914240. Current Snapshot: MvccSnapshot[applied={T|T < 7333327434331398144}]
I20260925 19:05:43.153620 11018 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:05:43.153710 11018 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:05:43.154428 11018 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d66b43d9c8d44c0a95231c8ed1ddb61: Sending DeleteTablet for 3 replicas of tablet 3466ed3f6d94412e941cc0caa78f227e
I20260925 19:05:43.154852 11412 tablet_service.cc:1514] Processing DeleteTablet for tablet 3466ed3f6d94412e941cc0caa78f227e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:05:43 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:57888
I20260925 19:05:43.154877 11144 tablet_service.cc:1514] Processing DeleteTablet for tablet 3466ed3f6d94412e941cc0caa78f227e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:05:43 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:56098
I20260925 19:05:43.155079 11278 tablet_service.cc:1514] Processing DeleteTablet for tablet 3466ed3f6d94412e941cc0caa78f227e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:05:43 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:46284
I20260925 19:05:43.155094 11519 tablet_replica.cc:330] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a: stopping tablet replica
I20260925 19:05:43.155194 11519 raft_consensus.cc:2229] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:05:43.155283 11519 raft_consensus.cc:2258] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:05:43.155282 11520 tablet_replica.cc:330] T 3466ed3f6d94412e941cc0caa78f227e P 8d77eaa3f6af468187a99e3dbbf4954e: stopping tablet replica
I20260925 19:05:43.155438 11518 tablet_replica.cc:330] T 3466ed3f6d94412e941cc0caa78f227e P 69f01d88414e48719504a5395c560bc0: stopping tablet replica
I20260925 19:05:43.155532 11518 raft_consensus.cc:2229] T 3466ed3f6d94412e941cc0caa78f227e P 69f01d88414e48719504a5395c560bc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:05:43.155653 11518 raft_consensus.cc:2258] T 3466ed3f6d94412e941cc0caa78f227e P 69f01d88414e48719504a5395c560bc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:05:43.155691 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 11081
I20260925 19:05:43.155777 11519 ts_tablet_manager.cc:1905] T 3466ed3f6d94412e941cc0caa78f227e P c48d16162ccb4790a9c0761f4775cc8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:05:43.156329 11520 raft_consensus.cc:2229] T 3466ed3f6d94412e941cc0caa78f227e P 8d77eaa3f6af468187a99e3dbbf4954e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:05:43.156445 11520 raft_consensus.cc:2258] T 3466ed3f6d94412e941cc0caa78f227e P 8d77eaa3f6af468187a99e3dbbf4954e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:05:43.157214 11520 ts_tablet_manager.cc:1905] T 3466ed3f6d94412e941cc0caa78f227e P 8d77eaa3f6af468187a99e3dbbf4954e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:05:43.158100 11518 ts_tablet_manager.cc:1905] T 3466ed3f6d94412e941cc0caa78f227e P 69f01d88414e48719504a5395c560bc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:05:43.159201 11520 ts_tablet_manager.cc:1918] T 3466ed3f6d94412e941cc0caa78f227e P 8d77eaa3f6af468187a99e3dbbf4954e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:05:43.159286 11520 log.cc:1196] T 3466ed3f6d94412e941cc0caa78f227e P 8d77eaa3f6af468187a99e3dbbf4954e: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-1/wal/wals/3466ed3f6d94412e941cc0caa78f227e
I20260925 19:05:43.159544 11518 ts_tablet_manager.cc:1918] T 3466ed3f6d94412e941cc0caa78f227e P 69f01d88414e48719504a5395c560bc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:05:43.159576 11520 ts_tablet_manager.cc:1939] T 3466ed3f6d94412e941cc0caa78f227e P 8d77eaa3f6af468187a99e3dbbf4954e: Deleting consensus metadata
I20260925 19:05:43.159642 11518 log.cc:1196] T 3466ed3f6d94412e941cc0caa78f227e P 69f01d88414e48719504a5395c560bc0: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790363087086945-787-0/minicluster-data/ts-2/wal/wals/3466ed3f6d94412e941cc0caa78f227e
I20260925 19:05:43.159898 11518 ts_tablet_manager.cc:1939] T 3466ed3f6d94412e941cc0caa78f227e P 69f01d88414e48719504a5395c560bc0: Deleting consensus metadata
I20260925 19:05:43.159993 11003 catalog_manager.cc:4841] TS 8d77eaa3f6af468187a99e3dbbf4954e (127.0.196.194:42309): tablet 3466ed3f6d94412e941cc0caa78f227e (table test-table [id=d1bd05724692487ab95c193592d99f74]) successfully deleted
I20260925 19:05:43.160295 11005 catalog_manager.cc:4841] TS 69f01d88414e48719504a5395c560bc0 (127.0.196.195:43197): tablet 3466ed3f6d94412e941cc0caa78f227e (table test-table [id=d1bd05724692487ab95c193592d99f74]) successfully deleted
W20260925 19:05:43.161801 11006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.196.193:41175 (error 108)
I20260925 19:05:43.162015 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 11215
I20260925 19:05:43.162350 11490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:05:43.162226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:41175 (local address 127.0.0.1:56106)
0925 19:05:43.162279 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 19:05:43.162311 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.193:41175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 19:05:43.162460 11006 catalog_manager.cc:4584] TS c48d16162ccb4790a9c0761f4775cc8a (127.0.196.193:41175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3466ed3f6d94412e941cc0caa78f227e: Network error: Client connection negotiation failed: client connection to 127.0.196.193:41175: connect: Connection refused (error 111)
I20260925 19:05:43.167109 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 11349
I20260925 19:05:43.171609 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 10987
2026-09-25T19:05:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1927 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:05:43 dist-test-slave-rn98 krb5kdc[11524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:05:43 dist-test-slave-rn98 krb5kdc[11524](info): set up 2 sockets
Sep 25 19:05:43 dist-test-slave-rn98 krb5kdc[11524](info): commencing operation
krb5kdc: starting...
W20260925 19:05:44.223719 787 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:05:44 dist-test-slave-rn98 krb5kdc[11524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:05:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:05:44Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.196.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.196.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.196.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/rangeradmin_127.0.196.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.196.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/rangeradmin_127.0.196.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.196.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.196.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.196.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/rangerlookup_127.0.196.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.196.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/rangerlookup_127.0.196.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.196.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.196.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.196.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/HTTP_127.0.196.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.196.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/HTTP_127.0.196.212@KRBTEST.COM.keytab.
I20260925 19:05:44.390087 787 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskhGKzmd/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/postgres -l logfile start
2026-09-25 19:05:47.113 UTC [11554] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 19:05:47.113 UTC [11554] LOG: listening on IPv4 address "127.0.196.212", port 55895
2026-09-25 19:05:47.118 UTC [11554] LOG: listening on Unix socket "/tmp/.s.PGSQL.55895"
2026-09-25 19:05:47.157 UTC [11557] LOG: database system was shut down at 2026-09-25 19:05:45 UTC
2026-09-25 19:05:47.161 UTC [11554] LOG: database system is ready to accept connections
I20260925 19:05:48.127540 787 mini_postgres.cc:96] Postgres bound to 55895
2026-09-25 19:05:48.133 UTC [11565] FATAL: database "slave" does not exist
127.0.196.212:55895 - accepting connections
I20260925 19:05:48.134130 787 mini_ranger.cc:161] Starting Ranger...
I20260925 19:05:48.149544 787 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 19:05:48.686949 787 mini_ranger.cc:87] Created ranger Postgres database
I20260925 19:05:48.687036 787 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ranger-admin
2026-09-25 19:05:48,957 [I] DB FLAVOR :POSTGRES
2026-09-25 19:05:48,958 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 19:05:48,958 [I] Checking connection..
2026-09-25 19:05:48,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:55895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 19:05:49,198 [I] Checking connection passed.
2026-09-25 19:05:49,199 [I] --------- Verifying version history table ---------
2026-09-25 19:05:49,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:55895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 19:05:49,467 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 19:05:49,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:55895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 19:05:49,726 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 19:05:49,727 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 19:05:49,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:55895/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 19:05:49.976 UTC [11629] WARNING: there is no transaction in progress
2026-09-25 19:05:49,982 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 19:05:49,982 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:55895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 19:05:50,253 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 19:05:50,253 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 19:05:50,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:55895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 19:05:50,487 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:55895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 19:05:50,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:55895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rn98.c.gcp-upstream.internal','N') ;"
2026-09-25 19:05:51,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:55895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 19:05:51,263 [I] Table x_portal_user does not exist in database ranger
2026-09-25 19:05:51,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:55895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 19:05:51,516 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 19:05:51,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:55895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 19:05:51,752 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:55895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 19:05:52,021 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 19:05:52,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:55895/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 19:05:53.067 UTC [11737] WARNING: there is no transaction in progress
2026-09-25 19:05:53.085 UTC [11737] WARNING: there is no transaction in progress
2026-09-25 19:05:53.103 UTC [11737] WARNING: there is no transaction in progress
2026-09-25 19:05:53.120 UTC [11737] WARNING: there is no transaction in progress
2026-09-25 19:05:53.138 UTC [11737] WARNING: there is no transaction in progress
2026-09-25 19:05:53.156 UTC [11737] WARNING: there is no transaction in progress
2026-09-25 19:05:53.215 UTC [11737] WARNING: there is no transaction in progress
2026-09-25 19:05:53.223 UTC [11737] WARNING: there is no transaction in progress
2026-09-25 19:05:53.239 UTC [11737] WARNING: there is no transaction in progress
2026-09-25 19:05:53.250 UTC [11737] WARNING: there is no transaction in progress
2026-09-25 19:05:53.261 UTC [11737] WARNING: there is no transaction in progress
2026-09-25 19:05:53.275 UTC [11737] WARNING: there is no transaction in progress
2026-09-25 19:05:53.287 UTC [11737] WARNING: there is no transaction in progress
2026-09-25 19:05:53.295 UTC [11737] WARNING: there is no transaction in progress
2026-09-25 19:05:53.302 UTC [11737] WARNING: there is no transaction in progress
2026-09-25 19:05:53.311 UTC [11737] WARNING: there is no transaction in progress
2026-09-25 19:05:54,031 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 19:05:54,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:55895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 19:05:54,273 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 19:05:54,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:55895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 19:05:54,525 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 19:05:54,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:55895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 19:05:54,804 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 19:05:54,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:55895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 19:05:55,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:55895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 19:05:55,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:55895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rn98.c.gcp-upstream.internal';"
2026-09-25 19:05:55,590 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 19:05:55,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:55895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 19:05:55,845 [I] DB_PATCHES have already been applied
I20260925 19:05:55.851656 787 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ranger-admin:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskhGKzmd/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 19:05:55.851763 787 mini_ranger.cc:193] Using host: 127.0.196.212
I20260925 19:05:55.854941 787 mini_ranger.cc:227] Ranger admin URL: http://127.0.196.212:50249
Sep 25, 2026 7:05:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 7:05:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 7:05:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 7:05:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 7:05:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 7:05:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 7:05:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 19:05:57.045188 787 mini_ranger.cc:160] Time spent starting Ranger: real 8.911s user 0.003s sys 0.006s
Sep 25, 2026 7:06:18 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 7:06:18 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 7:06:18 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 7:06:18 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 7:06:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 7:06:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 7:06:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 7:06:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 7:06:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 7:06:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 7:06:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 7:06:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 7:06:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 7:06:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 7:06:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 7:06:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 7:06:18 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 7:06:19 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 19:06:20.041746 787 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.196.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.196.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.196.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/rangerkms_127.0.196.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.196.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/rangerkms_127.0.196.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.196.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.196.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.196.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/HTTP_127.0.196.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.196.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/HTTP_127.0.196.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 19:06:20 dist-test-slave-rn98 krb5kdc[11524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363180, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 19:06:20.128595 787 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 19:06:20.138980 787 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 19:06:21.076786 787 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 19:06:21.076875 787 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ranger-kms
I20260925 19:06:21.076894 787 mini_ranger_kms.cc:227] Using postgres at 127.0.196.212:55895
2026-09-25 19:06:21,265 [I] DB FLAVOR :POSTGRES
2026-09-25 19:06:21,265 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 19:06:21,265 [I] Checking connection
2026-09-25 19:06:21,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:55895/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 19:06:21,529 [I] connection success
2026-09-25 19:06:21,529 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 19:06:21,529 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 19:06:21,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:55895/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 19:06:21,767 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 19:06:21,767 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 19:06:21,767 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 19:06:21,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:55895/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 19:06:22,051 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 19:06:22.523260 787 mini_ranger_kms.cc:310] Created kms service
I20260925 19:06:22.760116 787 mini_ranger_kms.cc:326] Created kudu user
I20260925 19:06:22.808970 787 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 19:06:22 dist-test-slave-rn98 krb5kdc[11524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363180, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.196.212@KRBTEST.COM
I20260925 19:06:23.241165 787 mini_ranger_kms.cc:384] Added ranger policy
I20260925 19:06:23.241284 787 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ranger-kms:/tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 19:06:23.241304 787 mini_ranger_kms.cc:240] Using host: 127.0.196.212
I20260925 19:06:23.245203 787 mini_ranger_kms.cc:276] Ranger KMS PID: 11918
I20260925 19:06:23.245337 787 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.196.212:56895
Sep 25, 2026 7:06:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 7:06:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 7:06:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 7:06:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 7:06:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 7:06:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 19:06:25.515892 787 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.387s user 0.005s sys 0.166s
I20260925 19:06:25.516006 787 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 19:06:25.516080 787 mini_ranger_kms.cc:401] 127.0.196.212:56895/kms/v1/keys
Sep 25 19:06:25 dist-test-slave-rn98 krb5kdc[11524](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790363185, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.196.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 19:06:26 dist-test-slave-rn98 krb5kdc[11524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363185, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.196.212@KRBTEST.COM for HTTP/127.0.196.212@KRBTEST.COM
Sep 25 19:06:26 dist-test-slave-rn98 krb5kdc[11524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363185, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.196.212@KRBTEST.COM for HTTP/127.0.196.212@KRBTEST.COM
Sep 25 19:06:28 dist-test-slave-rn98 krb5kdc[11524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:06:28.693852 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:42337
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:40491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:42337
--ranger_config_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.196.212:56895/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:06:28.791275 11968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:06:28.791514 11968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:06:28.791541 11968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:06:28.792712 11968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:06:28.792754 11968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:06:28.792768 11968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:06:28.792781 11968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:06:28.792793 11968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:06:28.794304 11968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:40491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.196.212:56895/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:42337
--ranger_config_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:42337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:06:28.794495 11968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:06:28.794739 11968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:06:28.800197 11974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:06:28.800282 11968 server_base.cc:1034] running on GCE node
W20260925 19:06:28.800244 11976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:06:28.800249 11973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:06:28.800741 11968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:06:28.801074 11968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:06:28.802243 11968 hybrid_clock.cc:648] HybridClock initialized: now 1790363188802222 us; error 44 us; skew 500 ppm
Sep 25 19:06:28 dist-test-slave-rn98 krb5kdc[11524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:06:28.805085 11968 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:06:28.805599 11968 webserver.cc:460] Webserver started at http://127.0.196.254:33103/ using document root <none> and password file <none>
I20260925 19:06:28.805832 11968 fs_manager.cc:359] Metadata directory not provided
I20260925 19:06:28.805891 11968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:06:28.806037 11968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 19:06:28 dist-test-slave-rn98 krb5kdc[11524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for HTTP/127.0.196.212@KRBTEST.COM
I20260925 19:06:28.866245 11968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "1fcd20a701c843c69d6d455ab51f9ad3"
format_stamp: "Formatted at 2026-09-25 19:06:28 on dist-test-slave-rn98"
server_key: "be117bbeaaefb30951614598b6600d7f"
server_key_iv: "c324897f7ae4880902149fc4d8ca10c6"
server_key_version: "kuduclusterkey@0"
I20260925 19:06:28.866762 11968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "1fcd20a701c843c69d6d455ab51f9ad3"
format_stamp: "Formatted at 2026-09-25 19:06:28 on dist-test-slave-rn98"
server_key: "be117bbeaaefb30951614598b6600d7f"
server_key_iv: "c324897f7ae4880902149fc4d8ca10c6"
server_key_version: "kuduclusterkey@0"
I20260925 19:06:28.868566 11968 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 19:06:28.883929 11983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:06:28.884166 11968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:06:28.884265 11968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "1fcd20a701c843c69d6d455ab51f9ad3"
format_stamp: "Formatted at 2026-09-25 19:06:28 on dist-test-slave-rn98"
server_key: "be117bbeaaefb30951614598b6600d7f"
server_key_iv: "c324897f7ae4880902149fc4d8ca10c6"
server_key_version: "kuduclusterkey@0"
I20260925 19:06:28.884370 11968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:06:28.903591 11968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:06:28.905828 11968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:06:28.906024 11968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:06:28.911662 11968 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:42337
I20260925 19:06:28.911732 12035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:42337 every 8 connection(s)
I20260925 19:06:28.912133 11968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:06:28.912846 12036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:06:28.915421 12036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fcd20a701c843c69d6d455ab51f9ad3: Bootstrap starting.
I20260925 19:06:28.916122 12036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fcd20a701c843c69d6d455ab51f9ad3: Neither blocks nor log segments found. Creating new log.
I20260925 19:06:28.916443 12036 log.cc:824] T 00000000000000000000000000000000 P 1fcd20a701c843c69d6d455ab51f9ad3: Log is configured to *not* fsync() on all Append() calls
I20260925 19:06:28.917208 12036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fcd20a701c843c69d6d455ab51f9ad3: No bootstrap required, opened a new log
I20260925 19:06:28.919448 12036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fcd20a701c843c69d6d455ab51f9ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fcd20a701c843c69d6d455ab51f9ad3" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 42337 } }
I20260925 19:06:28.919618 12036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fcd20a701c843c69d6d455ab51f9ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:06:28.919642 12036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fcd20a701c843c69d6d455ab51f9ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fcd20a701c843c69d6d455ab51f9ad3, State: Initialized, Role: FOLLOWER
I20260925 19:06:28.919811 12036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fcd20a701c843c69d6d455ab51f9ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fcd20a701c843c69d6d455ab51f9ad3" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 42337 } }
I20260925 19:06:28.919900 12036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fcd20a701c843c69d6d455ab51f9ad3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:06:28.919945 12036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fcd20a701c843c69d6d455ab51f9ad3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:06:28.919996 12036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fcd20a701c843c69d6d455ab51f9ad3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:06:28.920612 12036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fcd20a701c843c69d6d455ab51f9ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fcd20a701c843c69d6d455ab51f9ad3" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 42337 } }
I20260925 19:06:28.920737 12036 leader_election.cc:302] T 00000000000000000000000000000000 P 1fcd20a701c843c69d6d455ab51f9ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fcd20a701c843c69d6d455ab51f9ad3; no voters:
I20260925 19:06:28.920977 12036 leader_election.cc:288] T 00000000000000000000000000000000 P 1fcd20a701c843c69d6d455ab51f9ad3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:06:28.921011 12039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fcd20a701c843c69d6d455ab51f9ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:06:28.921100 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 11968
I20260925 19:06:28.921196 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:06:28.921250 12039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fcd20a701c843c69d6d455ab51f9ad3 [term 1 LEADER]: Becoming Leader. State: Replica: 1fcd20a701c843c69d6d455ab51f9ad3, State: Running, Role: LEADER
I20260925 19:06:28.921358 787 external_mini_cluster.cc:1428] Setting key 943b519480c599237b4b6fb29c4a2755
I20260925 19:06:28.921339 12039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fcd20a701c843c69d6d455ab51f9ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fcd20a701c843c69d6d455ab51f9ad3" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 42337 } }
I20260925 19:06:28.921372 12036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fcd20a701c843c69d6d455ab51f9ad3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:06:28.921645 12039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcd20a701c843c69d6d455ab51f9ad3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fcd20a701c843c69d6d455ab51f9ad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fcd20a701c843c69d6d455ab51f9ad3" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 42337 } } }
I20260925 19:06:28.921730 12039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcd20a701c843c69d6d455ab51f9ad3 [sys.catalog]: This master's current role is: LEADER
I20260925 19:06:28.921844 12041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcd20a701c843c69d6d455ab51f9ad3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fcd20a701c843c69d6d455ab51f9ad3. Latest consensus state: current_term: 1 leader_uuid: "1fcd20a701c843c69d6d455ab51f9ad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fcd20a701c843c69d6d455ab51f9ad3" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 42337 } } }
I20260925 19:06:28.921972 12041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcd20a701c843c69d6d455ab51f9ad3 [sys.catalog]: This master's current role is: LEADER
I20260925 19:06:28.923653 12043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:06:28.924278 12043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:06:28.924661 12036 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 25 19:06:28 dist-test-slave-rn98 krb5kdc[11524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:06:28.934136 12043 catalog_manager.cc:1347] Generated new cluster ID: 91064b174fb3401fbb5a3b853940f80a
I20260925 19:06:28.934281 12043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:06:28.936655 12045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:06:28.922054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59196 (local address 127.0.196.254:42337)
0925 19:06:28.922233 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 19:06:28.922245 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 19:06:28.922289 (+ 44us) server_negotiation.cc:407] Connection header received
0925 19:06:28.922422 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:06:28.922429 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:06:28.922491 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:06:28.922707 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:06:28.923573 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:06:28.925120 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:06:28.926941 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:06:28.928454 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:06:28.928643 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:06:28.932081 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:06:28.932086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:06:28.932086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:06:28.932102 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:06:28.934835 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:06:28.935296 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:06:28.935301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:06:28.935301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:06:28.935426 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:06:28.935597 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:06:28.935597 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:06:28.935598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:06:28.935958 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:06:28.936120 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 19:06:28.936239 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 19:06:28.936369 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260925 19:06:28.941483 12043 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:06:28.942317 12043 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:06:28.959909 12043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fcd20a701c843c69d6d455ab51f9ad3: Generated new TSK 0
I20260925 19:06:28.960238 12043 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
I20260925 19:06:29.744084 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.196.212:56895/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:0
--local_ip_for_outbound_sockets=127.0.196.193
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:42337
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:40491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:06:29.871426 12081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:06:29.871748 12081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:06:29.871802 12081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:06:29.873769 12081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:06:29.873847 12081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:06:29.873894 12081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:06:29.876341 12081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:40491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.196.212:56895/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:42337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:06:29.876648 12081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:06:29.877017 12081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:06:29.883273 12089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:06:29.883524 12086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:06:29.883391 12081 server_base.cc:1034] running on GCE node
W20260925 19:06:29.883370 12087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:06:29.883867 12081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:06:29.884199 12081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:06:29.885387 12081 hybrid_clock.cc:648] HybridClock initialized: now 1790363189885371 us; error 32 us; skew 500 ppm
Sep 25 19:06:29 dist-test-slave-rn98 krb5kdc[11524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:06:29.888391 12081 init.cc:377] Logged in from keytab as kudu/127.0.196.193@KRBTEST.COM (short username kudu)
I20260925 19:06:29.889001 12081 webserver.cc:460] Webserver started at http://127.0.196.193:33065/ using document root <none> and password file <none>
I20260925 19:06:29.889308 12081 fs_manager.cc:359] Metadata directory not provided
I20260925 19:06:29.889376 12081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:06:29.889559 12081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 19:06:29 dist-test-slave-rn98 krb5kdc[11524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for HTTP/127.0.196.212@KRBTEST.COM
I20260925 19:06:29.903537 12081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-0/data/instance:
uuid: "9f02e7ecbe8b4b16b640ad065d01f0ee"
format_stamp: "Formatted at 2026-09-25 19:06:29 on dist-test-slave-rn98"
server_key: "968b251d0b62fa348ac2ea5227fbcbbf"
server_key_iv: "f771cb74321d1bc79c9cb4042d029635"
server_key_version: "kuduclusterkey@0"
I20260925 19:06:29.904029 12081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-0/wal/instance:
uuid: "9f02e7ecbe8b4b16b640ad065d01f0ee"
format_stamp: "Formatted at 2026-09-25 19:06:29 on dist-test-slave-rn98"
server_key: "968b251d0b62fa348ac2ea5227fbcbbf"
server_key_iv: "f771cb74321d1bc79c9cb4042d029635"
server_key_version: "kuduclusterkey@0"
I20260925 19:06:29.907016 12081 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 19:06:29.919433 12096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:06:29.922806 12081 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260925 19:06:29.922952 12081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "9f02e7ecbe8b4b16b640ad065d01f0ee"
format_stamp: "Formatted at 2026-09-25 19:06:29 on dist-test-slave-rn98"
server_key: "968b251d0b62fa348ac2ea5227fbcbbf"
server_key_iv: "f771cb74321d1bc79c9cb4042d029635"
server_key_version: "kuduclusterkey@0"
I20260925 19:06:29.923079 12081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:06:29.955749 12081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:06:29.958060 12081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:06:29.958264 12081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:06:29.958544 12081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:06:29.958967 12081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:06:29.959043 12081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:06:29.959110 12081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:06:29.959161 12081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:06:29.967146 12081 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:45015
I20260925 19:06:29.967543 12081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
I20260925 19:06:29.968125 12211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:45015 every 8 connection(s)
I20260925 19:06:29.971846 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 12081
I20260925 19:06:29.971943 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-0/wal/instance
I20260925 19:06:29.972151 787 external_mini_cluster.cc:1428] Setting key bca10f372148d01ea0e8c0780dd1e195
Sep 25 19:06:29 dist-test-slave-rn98 krb5kdc[11524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:06:29.978672 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:06:29.968624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:60843 (local address 127.0.196.254:42337)
0925 19:06:29.968743 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 19:06:29.968746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:06:29.968756 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:06:29.969239 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:06:29.969241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:06:29.969253 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:06:29.969299 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:06:29.969881 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:06:29.970408 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:06:29.971525 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:06:29.971877 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:06:29.971955 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:06:29.974862 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:06:29.974864 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:06:29.974865 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:06:29.974874 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:06:29.976685 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:06:29.977276 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:06:29.977278 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:06:29.977279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:06:29.977353 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:06:29.977760 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:06:29.977762 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:06:29.977762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:06:29.977927 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:06:29.978011 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 19:06:29.978427 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 19:06:29.978530 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260925 19:06:29.978912 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:06:29.968308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:42337 (local address 127.0.196.193:60843)
0925 19:06:29.968529 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 19:06:29.968577 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 19:06:29.969114 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:06:29.969386 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:06:29.969389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:06:29.969584 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:06:29.969780 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:06:29.969785 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:06:29.970513 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:06:29.970514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:06:29.971428 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:06:29.971430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:06:29.972037 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:06:29.972038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:06:29.972196 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:06:29.972955 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:06:29.972967 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:06:29.974754 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:06:29.976794 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:06:29.976801 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:06:29.976804 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:06:29.977175 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:06:29.977441 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:06:29.977442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:06:29.977442 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:06:29.977647 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:06:29.978017 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:06:29.978024 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:06:29.978237 (+ 213us) client_negotiation.cc:769] Sending connection context
0925 19:06:29.978376 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 19:06:29.978573 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":104,"threads_started":1}
I20260925 19:06:29.979622 12212 heartbeater.cc:347] Connected to a master server at 127.0.196.254:42337
I20260925 19:06:29.980123 12212 heartbeater.cc:464] Registering TS with master...
I20260925 19:06:29.980334 12212 heartbeater.cc:511] Master 127.0.196.254:42337 requested a full tablet report, sending...
I20260925 19:06:29.980967 12000 ts_manager.cc:194] Registered new tserver with Master: 9f02e7ecbe8b4b16b640ad065d01f0ee (127.0.196.193:45015)
WARNING: no policy specified for kudu/127.0.196.194@KRBTEST.COM; defaulting to no policy
I20260925 19:06:29.981889 12000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.193@KRBTEST.COM'} at 127.0.196.193:60843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
I20260925 19:06:30.000026 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.196.212:56895/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:0
--local_ip_for_outbound_sockets=127.0.196.194
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:42337
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:40491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 19:06:30 dist-test-slave-rn98 krb5kdc[11524](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790363190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 19:06:30.124871 12218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:06:30.125197 12218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:06:30.125247 12218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:06:30.127300 12218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:06:30.127416 12218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:06:30.127482 12218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:06:30.130075 12218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:40491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.196.212:56895/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:42337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:06:30.130541 12218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:06:30.130909 12218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:06:30.138401 12228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:06:30.138501 12225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:06:30.138406 12226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:06:30.139624 12218 server_base.cc:1034] running on GCE node
I20260925 19:06:30.139966 12218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:06:30.140266 12218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:06:30.141454 12218 hybrid_clock.cc:648] HybridClock initialized: now 1790363190141441 us; error 51 us; skew 500 ppm
Sep 25 19:06:30 dist-test-slave-rn98 krb5kdc[11524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:06:30.144840 12218 init.cc:377] Logged in from keytab as kudu/127.0.196.194@KRBTEST.COM (short username kudu)
I20260925 19:06:30.145378 12218 webserver.cc:460] Webserver started at http://127.0.196.194:35391/ using document root <none> and password file <none>
I20260925 19:06:30.145696 12218 fs_manager.cc:359] Metadata directory not provided
I20260925 19:06:30.145807 12218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:06:30.145953 12218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 19:06:30 dist-test-slave-rn98 krb5kdc[11524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for HTTP/127.0.196.212@KRBTEST.COM
I20260925 19:06:30.160611 12218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-1/data/instance:
uuid: "0c370ec2c0a54019a13fb41bc42e60e9"
format_stamp: "Formatted at 2026-09-25 19:06:30 on dist-test-slave-rn98"
server_key: "9f76c6143eadad1f1edf83e340ca6c28"
server_key_iv: "3170c8092ffc094b978b8189d753c92f"
server_key_version: "kuduclusterkey@0"
I20260925 19:06:30.161074 12218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-1/wal/instance:
uuid: "0c370ec2c0a54019a13fb41bc42e60e9"
format_stamp: "Formatted at 2026-09-25 19:06:30 on dist-test-slave-rn98"
server_key: "9f76c6143eadad1f1edf83e340ca6c28"
server_key_iv: "3170c8092ffc094b978b8189d753c92f"
server_key_version: "kuduclusterkey@0"
I20260925 19:06:30.166920 12218 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.000s
I20260925 19:06:30.183331 12236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:06:30.184406 12218 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260925 19:06:30.184527 12218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "0c370ec2c0a54019a13fb41bc42e60e9"
format_stamp: "Formatted at 2026-09-25 19:06:30 on dist-test-slave-rn98"
server_key: "9f76c6143eadad1f1edf83e340ca6c28"
server_key_iv: "3170c8092ffc094b978b8189d753c92f"
server_key_version: "kuduclusterkey@0"
I20260925 19:06:30.184634 12218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:06:30.213641 12218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:06:30.217303 12218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:06:30.217478 12218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:06:30.217743 12218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:06:30.218161 12218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:06:30.218242 12218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:06:30.218288 12218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:06:30.218307 12218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:06:30.227087 12218 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:41451
I20260925 19:06:30.227142 12349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:41451 every 8 connection(s)
I20260925 19:06:30.227638 12218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:06:30 dist-test-slave-rn98 krb5kdc[11524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:06:30.237797 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 12218
I20260925 19:06:30.237905 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-1/wal/instance
I20260925 19:06:30.238106 787 external_mini_cluster.cc:1428] Setting key b55cec3e1487873534f5a9c96ae04602
I20260925 19:06:30.242833 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:06:30.228441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:34289 (local address 127.0.196.254:42337)
0925 19:06:30.228492 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 19:06:30.228494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:06:30.228865 (+ 371us) server_negotiation.cc:407] Connection header received
0925 19:06:30.229710 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:06:30.229712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:06:30.229735 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:06:30.229816 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:06:30.230676 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.231494 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.233205 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.233529 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.233605 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:06:30.236448 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:06:30.236450 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:06:30.236450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:06:30.236459 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:06:30.239294 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:06:30.241507 (+ 2213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:06:30.241509 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:06:30.241509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:06:30.241580 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:06:30.241974 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:06:30.241975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:06:30.241975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:06:30.242127 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:06:30.242171 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 19:06:30.242568 (+ 397us) server_negotiation.cc:299] Negotiation successful
0925 19:06:30.242701 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:06:30.243659 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:06:30.228417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:42337 (local address 127.0.196.194:34289)
0925 19:06:30.228696 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 19:06:30.228742 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 19:06:30.229573 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:06:30.229960 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:06:30.229965 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:06:30.230226 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:06:30.230518 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:06:30.230524 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.231615 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.231616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:06:30.233098 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:06:30.233101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.233681 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.233682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:06:30.233854 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:06:30.234514 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:06:30.234524 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:06:30.236316 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:06:30.241007 (+ 4691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:06:30.241021 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:06:30.241025 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:06:30.241400 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:06:30.241664 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:06:30.241665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:06:30.241666 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:06:30.241894 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:06:30.242217 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:06:30.242223 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:06:30.242411 (+ 188us) client_negotiation.cc:769] Sending connection context
0925 19:06:30.243232 (+ 821us) client_negotiation.cc:240] Negotiation successful
0925 19:06:30.243421 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":77,"threads_started":1}
I20260925 19:06:30.244532 12350 heartbeater.cc:347] Connected to a master server at 127.0.196.254:42337
I20260925 19:06:30.244632 12350 heartbeater.cc:464] Registering TS with master...
I20260925 19:06:30.244849 12350 heartbeater.cc:511] Master 127.0.196.254:42337 requested a full tablet report, sending...
I20260925 19:06:30.245282 12000 ts_manager.cc:194] Registered new tserver with Master: 0c370ec2c0a54019a13fb41bc42e60e9 (127.0.196.194:41451)
I20260925 19:06:30.245790 12000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.194@KRBTEST.COM'} at 127.0.196.194:34289
WARNING: no policy specified for kudu/127.0.196.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
I20260925 19:06:30.276131 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.196.212:56895/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:0
--local_ip_for_outbound_sockets=127.0.196.195
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:42337
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:40491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:06:30.404695 12355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:06:30.405071 12355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:06:30.405158 12355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:06:30.407073 12355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:06:30.407207 12355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:06:30.407313 12355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:06:30.409651 12355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:40491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.196.212:56895/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:42337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:06:30.410187 12355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:06:30.410558 12355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:06:30.423354 12360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:06:30.423506 12363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:06:30.423377 12361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:06:30.424539 12355 server_base.cc:1034] running on GCE node
I20260925 19:06:30.424787 12355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:06:30.425071 12355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:06:30.426251 12355 hybrid_clock.cc:648] HybridClock initialized: now 1790363190426222 us; error 42 us; skew 500 ppm
Sep 25 19:06:30 dist-test-slave-rn98 krb5kdc[11524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:06:30.429579 12355 init.cc:377] Logged in from keytab as kudu/127.0.196.195@KRBTEST.COM (short username kudu)
I20260925 19:06:30.430174 12355 webserver.cc:460] Webserver started at http://127.0.196.195:38281/ using document root <none> and password file <none>
I20260925 19:06:30.430461 12355 fs_manager.cc:359] Metadata directory not provided
I20260925 19:06:30.430523 12355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:06:30.430671 12355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 19:06:30 dist-test-slave-rn98 krb5kdc[11524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for HTTP/127.0.196.212@KRBTEST.COM
I20260925 19:06:30.444509 12355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-2/data/instance:
uuid: "5865c945f2284783867b927b15f11b39"
format_stamp: "Formatted at 2026-09-25 19:06:30 on dist-test-slave-rn98"
server_key: "82a715c22e469362b7dd5ad61fc02c98"
server_key_iv: "a4660a669fcb99b475ef177fe29a4918"
server_key_version: "kuduclusterkey@0"
I20260925 19:06:30.444981 12355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-2/wal/instance:
uuid: "5865c945f2284783867b927b15f11b39"
format_stamp: "Formatted at 2026-09-25 19:06:30 on dist-test-slave-rn98"
server_key: "82a715c22e469362b7dd5ad61fc02c98"
server_key_iv: "a4660a669fcb99b475ef177fe29a4918"
server_key_version: "kuduclusterkey@0"
I20260925 19:06:30.447160 12355 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 19:06:30.469421 12370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:06:30.469702 12355 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 19:06:30.469818 12355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "5865c945f2284783867b927b15f11b39"
format_stamp: "Formatted at 2026-09-25 19:06:30 on dist-test-slave-rn98"
server_key: "82a715c22e469362b7dd5ad61fc02c98"
server_key_iv: "a4660a669fcb99b475ef177fe29a4918"
server_key_version: "kuduclusterkey@0"
I20260925 19:06:30.469933 12355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:06:30.495684 12355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:06:30.498296 12355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:06:30.498520 12355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:06:30.499030 12355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:06:30.499480 12355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:06:30.499570 12355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:06:30.499651 12355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:06:30.499711 12355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:06:30.510979 12355 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:35413
I20260925 19:06:30.511574 12355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
I20260925 19:06:30.512530 12483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:35413 every 8 connection(s)
I20260925 19:06:30.514390 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 12355
I20260925 19:06:30.514500 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-2/wal/instance
I20260925 19:06:30.514734 787 external_mini_cluster.cc:1428] Setting key a88d3fe8046cb9489df770fc35ea06b2
Sep 25 19:06:30 dist-test-slave-rn98 krb5kdc[11524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:06:30.525336 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:06:30.512791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:48097 (local address 127.0.196.254:42337)
0925 19:06:30.512842 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 19:06:30.512844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:06:30.513109 (+ 265us) server_negotiation.cc:407] Connection header received
0925 19:06:30.513828 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:06:30.513830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:06:30.513844 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:06:30.513878 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:06:30.515834 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.516674 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.518281 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.518556 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.518610 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:06:30.521482 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:06:30.521484 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:06:30.521485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:06:30.521494 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:06:30.523440 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:06:30.524118 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:06:30.524119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:06:30.524120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:06:30.524184 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:06:30.524497 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:06:30.524499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:06:30.524499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:06:30.524649 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:06:30.524698 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 19:06:30.525140 (+ 442us) server_negotiation.cc:299] Negotiation successful
0925 19:06:30.525231 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:06:30.526913 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:06:30.512795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:42337 (local address 127.0.196.195:48097)
0925 19:06:30.512963 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 19:06:30.513015 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 19:06:30.513679 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:06:30.514085 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:06:30.514090 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:06:30.514351 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:06:30.515223 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:06:30.515231 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.516807 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.516808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:06:30.518160 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:06:30.518164 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.518734 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.518735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:06:30.518895 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:06:30.519555 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:06:30.519567 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:06:30.521349 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:06:30.523556 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:06:30.523565 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:06:30.523569 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:06:30.524006 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:06:30.524257 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:06:30.524258 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:06:30.524258 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:06:30.524412 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:06:30.524751 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:06:30.524758 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:06:30.524946 (+ 188us) client_negotiation.cc:769] Sending connection context
0925 19:06:30.525728 (+ 782us) client_negotiation.cc:240] Negotiation successful
0925 19:06:30.525921 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260925 19:06:30.527709 12484 heartbeater.cc:347] Connected to a master server at 127.0.196.254:42337
I20260925 19:06:30.527817 12484 heartbeater.cc:464] Registering TS with master...
I20260925 19:06:30.528019 12484 heartbeater.cc:511] Master 127.0.196.254:42337 requested a full tablet report, sending...
I20260925 19:06:30.528533 12000 ts_manager.cc:194] Registered new tserver with Master: 5865c945f2284783867b927b15f11b39 (127.0.196.195:35413)
I20260925 19:06:30.529048 12000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.195@KRBTEST.COM'} at 127.0.196.195:48097
I20260925 19:06:30.539131 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:06:30.550978 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:06:30.541034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59210 (local address 127.0.196.254:42337)
0925 19:06:30.541084 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 19:06:30.541086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:06:30.541225 (+ 139us) server_negotiation.cc:407] Connection header received
0925 19:06:30.541339 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:06:30.541340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:06:30.541351 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:06:30.541384 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:06:30.542243 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.543018 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.545676 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.545998 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.546045 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:06:30.547813 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:06:30.547815 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:06:30.547815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:06:30.547824 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:06:30.549582 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:06:30.550046 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:06:30.550047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:06:30.550048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:06:30.550110 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:06:30.550331 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:06:30.550332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:06:30.550332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:06:30.550467 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:06:30.550542 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 19:06:30.550791 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 19:06:30.550877 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:06:30.554804 12000 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:06:30.557132 12000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:06:30.572312 12496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:06:30.565973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:41451 (local address 127.0.0.1:42294)
0925 19:06:30.567065 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0925 19:06:30.567073 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 19:06:30.567111 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:06:30.567414 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:06:30.567418 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:06:30.567420 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:06:30.567495 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:06:30.567497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.569067 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.569068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:06:30.570809 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:06:30.570813 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.571962 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.571964 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:06:30.572048 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:06:30.572065 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 19:06:30.572141 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 19:06:30.572219 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1078,"thread_start_us":79,"threads_started":1}
I20260925 19:06:30.572799 12501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:06:30.566169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42294 (local address 127.0.196.194:41451)
0925 19:06:30.566890 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0925 19:06:30.566895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:06:30.567163 (+ 268us) server_negotiation.cc:407] Connection header received
0925 19:06:30.567222 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:06:30.567225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:06:30.567241 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:06:30.567314 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:06:30.567577 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.568971 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.570923 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.571749 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.571832 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:06:30.571916 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 19:06:30.572666 (+ 750us) server_negotiation.cc:299] Negotiation successful
0925 19:06:30.572715 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":95,"threads_started":1}
I20260925 19:06:30.574193 12284 tablet_service.cc:1467] Processing CreateTablet for tablet ebe2a194a7464453a5f7cd4fa8b3dd83 (DEFAULT_TABLE table=test-table [id=d04f516eea2a4cdda6cf2b3ca71332f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:06:30.574633 12284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe2a194a7464453a5f7cd4fa8b3dd83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:06:30.579129 12498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:06:30.565005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:45015 (local address 127.0.0.1:34702)
0925 19:06:30.566577 (+ 1572us) negotiation.cc:107] Waiting for socket to connect
0925 19:06:30.566668 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0925 19:06:30.566739 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:06:30.575419 (+ 8680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:06:30.575420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:06:30.575423 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:06:30.575484 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:06:30.575485 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.576744 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.576745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:06:30.577912 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:06:30.577915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.578886 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.578887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:06:30.578962 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:06:30.578968 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 19:06:30.578999 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 19:06:30.579043 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1534,"thread_start_us":86,"threads_started":1}
I20260925 19:06:30.579630 12500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:06:30.565252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:35413 (local address 127.0.0.1:58466)
0925 19:06:30.566941 (+ 1689us) negotiation.cc:107] Waiting for socket to connect
0925 19:06:30.566951 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:06:30.566992 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:06:30.574949 (+ 7957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:06:30.574950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:06:30.574954 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:06:30.575048 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:06:30.575050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.576450 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.576452 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:06:30.577946 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:06:30.577950 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.579391 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.579393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:06:30.579458 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:06:30.579463 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:06:30.579513 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 19:06:30.579555 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1675,"thread_start_us":59,"threads_started":1}
I20260925 19:06:30.579797 12497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:06:30.565110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34702 (local address 127.0.196.193:45015)
0925 19:06:30.575183 (+ 10073us) server_negotiation.cc:206] Beginning negotiation
0925 19:06:30.575189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:06:30.575201 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:06:30.575235 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:06:30.575237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:06:30.575255 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:06:30.575341 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:06:30.575559 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.576632 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.578029 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.578782 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.579555 (+ 773us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:06:30.579647 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 19:06:30.579675 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 19:06:30.579719 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10010,"thread_start_us":72,"threads_started":1}
I20260925 19:06:30.580264 12499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:06:30.565786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58466 (local address 127.0.196.195:35413)
0925 19:06:30.574750 (+ 8964us) server_negotiation.cc:206] Beginning negotiation
0925 19:06:30.574756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:06:30.574770 (+ 14us) server_negotiation.cc:407] Connection header received
0925 19:06:30.574815 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:06:30.574817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:06:30.574834 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:06:30.574934 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:06:30.575169 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.576341 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.578080 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.579183 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.579273 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:06:30.579355 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 19:06:30.580111 (+ 756us) server_negotiation.cc:299] Negotiation successful
0925 19:06:30.580160 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8915,"thread_start_us":84,"threads_started":1}
I20260925 19:06:30.580485 12146 tablet_service.cc:1467] Processing CreateTablet for tablet ebe2a194a7464453a5f7cd4fa8b3dd83 (DEFAULT_TABLE table=test-table [id=d04f516eea2a4cdda6cf2b3ca71332f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:06:30.580850 12146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe2a194a7464453a5f7cd4fa8b3dd83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:06:30.581046 12418 tablet_service.cc:1467] Processing CreateTablet for tablet ebe2a194a7464453a5f7cd4fa8b3dd83 (DEFAULT_TABLE table=test-table [id=d04f516eea2a4cdda6cf2b3ca71332f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:06:30.581394 12418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe2a194a7464453a5f7cd4fa8b3dd83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:06:30.585570 12504 tablet_bootstrap.cc:492] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39: Bootstrap starting.
I20260925 19:06:30.586872 12504 tablet_bootstrap.cc:654] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39: Neither blocks nor log segments found. Creating new log.
I20260925 19:06:30.587321 12504 log.cc:824] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39: Log is configured to *not* fsync() on all Append() calls
I20260925 19:06:30.589030 12504 tablet_bootstrap.cc:492] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39: No bootstrap required, opened a new log
I20260925 19:06:30.589115 12504 ts_tablet_manager.cc:1397] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 19:06:30.590835 12502 tablet_bootstrap.cc:492] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 0c370ec2c0a54019a13fb41bc42e60e9: Bootstrap starting.
I20260925 19:06:30.591861 12504 raft_consensus.cc:348] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f02e7ecbe8b4b16b640ad065d01f0ee" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45015 } } peers { permanent_uuid: "5865c945f2284783867b927b15f11b39" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35413 } } peers { permanent_uuid: "0c370ec2c0a54019a13fb41bc42e60e9" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 41451 } }
I20260925 19:06:30.592051 12504 raft_consensus.cc:374] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:06:30.592083 12504 raft_consensus.cc:729] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5865c945f2284783867b927b15f11b39, State: Initialized, Role: FOLLOWER
I20260925 19:06:30.592128 12502 tablet_bootstrap.cc:654] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 0c370ec2c0a54019a13fb41bc42e60e9: Neither blocks nor log segments found. Creating new log.
I20260925 19:06:30.592475 12502 log.cc:824] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 0c370ec2c0a54019a13fb41bc42e60e9: Log is configured to *not* fsync() on all Append() calls
I20260925 19:06:30.592643 12504 consensus_queue.cc:260] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f02e7ecbe8b4b16b640ad065d01f0ee" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45015 } } peers { permanent_uuid: "5865c945f2284783867b927b15f11b39" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35413 } } peers { permanent_uuid: "0c370ec2c0a54019a13fb41bc42e60e9" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 41451 } }
I20260925 19:06:30.592947 12503 tablet_bootstrap.cc:492] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 9f02e7ecbe8b4b16b640ad065d01f0ee: Bootstrap starting.
I20260925 19:06:30.593132 12504 ts_tablet_manager.cc:1428] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260925 19:06:30.593359 12484 heartbeater.cc:503] Master 127.0.196.254:42337 was elected leader, sending a full tablet report...
I20260925 19:06:30.593811 12503 tablet_bootstrap.cc:654] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 9f02e7ecbe8b4b16b640ad065d01f0ee: Neither blocks nor log segments found. Creating new log.
I20260925 19:06:30.594156 12503 log.cc:824] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 9f02e7ecbe8b4b16b640ad065d01f0ee: Log is configured to *not* fsync() on all Append() calls
I20260925 19:06:30.596626 12502 tablet_bootstrap.cc:492] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 0c370ec2c0a54019a13fb41bc42e60e9: No bootstrap required, opened a new log
I20260925 19:06:30.596788 12502 ts_tablet_manager.cc:1397] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 0c370ec2c0a54019a13fb41bc42e60e9: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260925 19:06:30.599714 12502 raft_consensus.cc:348] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 0c370ec2c0a54019a13fb41bc42e60e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f02e7ecbe8b4b16b640ad065d01f0ee" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45015 } } peers { permanent_uuid: "5865c945f2284783867b927b15f11b39" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35413 } } peers { permanent_uuid: "0c370ec2c0a54019a13fb41bc42e60e9" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 41451 } }
I20260925 19:06:30.600034 12502 raft_consensus.cc:374] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 0c370ec2c0a54019a13fb41bc42e60e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:06:30.600139 12502 raft_consensus.cc:729] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 0c370ec2c0a54019a13fb41bc42e60e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c370ec2c0a54019a13fb41bc42e60e9, State: Initialized, Role: FOLLOWER
I20260925 19:06:30.600353 12502 consensus_queue.cc:260] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 0c370ec2c0a54019a13fb41bc42e60e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f02e7ecbe8b4b16b640ad065d01f0ee" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45015 } } peers { permanent_uuid: "5865c945f2284783867b927b15f11b39" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35413 } } peers { permanent_uuid: "0c370ec2c0a54019a13fb41bc42e60e9" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 41451 } }
I20260925 19:06:30.600750 12502 ts_tablet_manager.cc:1428] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 0c370ec2c0a54019a13fb41bc42e60e9: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 19:06:30.600958 12350 heartbeater.cc:503] Master 127.0.196.254:42337 was elected leader, sending a full tablet report...
I20260925 19:06:30.602368 12503 tablet_bootstrap.cc:492] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 9f02e7ecbe8b4b16b640ad065d01f0ee: No bootstrap required, opened a new log
I20260925 19:06:30.602455 12503 ts_tablet_manager.cc:1397] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 9f02e7ecbe8b4b16b640ad065d01f0ee: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260925 19:06:30.605204 12503 raft_consensus.cc:348] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 9f02e7ecbe8b4b16b640ad065d01f0ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f02e7ecbe8b4b16b640ad065d01f0ee" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45015 } } peers { permanent_uuid: "5865c945f2284783867b927b15f11b39" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35413 } } peers { permanent_uuid: "0c370ec2c0a54019a13fb41bc42e60e9" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 41451 } }
I20260925 19:06:30.605410 12503 raft_consensus.cc:374] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 9f02e7ecbe8b4b16b640ad065d01f0ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:06:30.605449 12503 raft_consensus.cc:729] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 9f02e7ecbe8b4b16b640ad065d01f0ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f02e7ecbe8b4b16b640ad065d01f0ee, State: Initialized, Role: FOLLOWER
I20260925 19:06:30.606177 12503 consensus_queue.cc:260] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 9f02e7ecbe8b4b16b640ad065d01f0ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f02e7ecbe8b4b16b640ad065d01f0ee" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45015 } } peers { permanent_uuid: "5865c945f2284783867b927b15f11b39" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35413 } } peers { permanent_uuid: "0c370ec2c0a54019a13fb41bc42e60e9" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 41451 } }
I20260925 19:06:30.606601 12503 ts_tablet_manager.cc:1428] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 9f02e7ecbe8b4b16b640ad065d01f0ee: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 19:06:30.606820 12212 heartbeater.cc:503] Master 127.0.196.254:42337 was elected leader, sending a full tablet report...
Sep 25 19:06:30 dist-test-slave-rn98 krb5kdc[11524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for HTTP/127.0.196.212@KRBTEST.COM
W20260925 19:06:30.718631 12213 tablet.cc:2369] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 9f02e7ecbe8b4b16b640ad065d01f0ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:06:30.730252 12351 tablet.cc:2369] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 0c370ec2c0a54019a13fb41bc42e60e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:06:30.762413 12485 tablet.cc:2369] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:06:30.871255 12507 raft_consensus.cc:482] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:06:30.871412 12507 raft_consensus.cc:504] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f02e7ecbe8b4b16b640ad065d01f0ee" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45015 } } peers { permanent_uuid: "5865c945f2284783867b927b15f11b39" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35413 } } peers { permanent_uuid: "0c370ec2c0a54019a13fb41bc42e60e9" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 41451 } }
I20260925 19:06:30.871817 12507 leader_election.cc:288] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f02e7ecbe8b4b16b640ad065d01f0ee (127.0.196.193:45015), 0c370ec2c0a54019a13fb41bc42e60e9 (127.0.196.194:41451)
I20260925 19:06:30.876045 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:06:30.872024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:45015 (local address 127.0.196.195:46693)
0925 19:06:30.872154 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 19:06:30.872168 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 19:06:30.872220 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:06:30.872529 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:06:30.872530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:06:30.872532 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:06:30.872606 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:06:30.872609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.873558 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.873559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:06:30.874812 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:06:30.874816 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.875725 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.875726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:06:30.875834 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:06:30.875839 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:06:30.875911 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 19:06:30.875959 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260925 19:06:30.876652 12497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:06:30.872328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:46693 (local address 127.0.196.193:45015)
0925 19:06:30.872380 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 19:06:30.872382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:06:30.872392 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:06:30.872422 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:06:30.872423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:06:30.872433 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:06:30.872505 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:06:30.872707 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.873458 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.874941 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.875485 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.875565 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:06:30.875669 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 19:06:30.876520 (+ 851us) server_negotiation.cc:299] Negotiation successful
0925 19:06:30.876576 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 19:06:30.877302 12148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe2a194a7464453a5f7cd4fa8b3dd83" candidate_uuid: "5865c945f2284783867b927b15f11b39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f02e7ecbe8b4b16b640ad065d01f0ee" is_pre_election: true
I20260925 19:06:30.877501 12148 raft_consensus.cc:2454] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 9f02e7ecbe8b4b16b640ad065d01f0ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5865c945f2284783867b927b15f11b39 in term 0.
I20260925 19:06:30.877810 12371 leader_election.cc:302] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5865c945f2284783867b927b15f11b39, 9f02e7ecbe8b4b16b640ad065d01f0ee; no voters:
I20260925 19:06:30.878036 12507 raft_consensus.cc:2781] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:06:30.878098 12507 raft_consensus.cc:482] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:06:30.878123 12507 raft_consensus.cc:3037] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:06:30.879140 12507 raft_consensus.cc:504] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f02e7ecbe8b4b16b640ad065d01f0ee" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45015 } } peers { permanent_uuid: "5865c945f2284783867b927b15f11b39" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35413 } } peers { permanent_uuid: "0c370ec2c0a54019a13fb41bc42e60e9" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 41451 } }
I20260925 19:06:30.879305 12507 leader_election.cc:288] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39 [CANDIDATE]: Term 1 election: Requested vote from peers 9f02e7ecbe8b4b16b640ad065d01f0ee (127.0.196.193:45015), 0c370ec2c0a54019a13fb41bc42e60e9 (127.0.196.194:41451)
I20260925 19:06:30.879645 12148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe2a194a7464453a5f7cd4fa8b3dd83" candidate_uuid: "5865c945f2284783867b927b15f11b39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f02e7ecbe8b4b16b640ad065d01f0ee"
I20260925 19:06:30.879730 12148 raft_consensus.cc:3037] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 9f02e7ecbe8b4b16b640ad065d01f0ee [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:06:30.880684 12148 raft_consensus.cc:2454] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 9f02e7ecbe8b4b16b640ad065d01f0ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5865c945f2284783867b927b15f11b39 in term 1.
I20260925 19:06:30.880949 12371 leader_election.cc:302] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5865c945f2284783867b927b15f11b39, 9f02e7ecbe8b4b16b640ad065d01f0ee; no voters:
I20260925 19:06:30.880964 12501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:06:30.872272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:40267 (local address 127.0.196.194:41451)
0925 19:06:30.872314 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:06:30.872316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:06:30.872531 (+ 215us) server_negotiation.cc:407] Connection header received
0925 19:06:30.872634 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:06:30.872636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:06:30.872646 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:06:30.872675 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:06:30.877378 (+ 4703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.878177 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.879732 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.880385 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.880442 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:06:30.880452 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:06:30.880840 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 19:06:30.880890 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:06:30.881158 12507 raft_consensus.cc:2781] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:06:30.881289 12511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:06:30.872027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:41451 (local address 127.0.196.195:40267)
0925 19:06:30.872468 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 19:06:30.872479 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:06:30.872559 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:06:30.877175 (+ 4616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:06:30.877177 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:06:30.877180 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:06:30.877268 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:06:30.877270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.878290 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.878291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:06:30.879610 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:06:30.879614 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.880661 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.880662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:06:30.880743 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:06:30.880748 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:06:30.881077 (+ 329us) client_negotiation.cc:240] Negotiation successful
0925 19:06:30.881138 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":11,"thread_start_us":118,"threads_started":1}
I20260925 19:06:30.881361 12507 raft_consensus.cc:686] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39 [term 1 LEADER]: Becoming Leader. State: Replica: 5865c945f2284783867b927b15f11b39, State: Running, Role: LEADER
I20260925 19:06:30.881462 12507 consensus_queue.cc:237] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f02e7ecbe8b4b16b640ad065d01f0ee" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45015 } } peers { permanent_uuid: "5865c945f2284783867b927b15f11b39" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35413 } } peers { permanent_uuid: "0c370ec2c0a54019a13fb41bc42e60e9" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 41451 } }
I20260925 19:06:30.881664 12304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe2a194a7464453a5f7cd4fa8b3dd83" candidate_uuid: "5865c945f2284783867b927b15f11b39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c370ec2c0a54019a13fb41bc42e60e9" is_pre_election: true
I20260925 19:06:30.881839 12304 raft_consensus.cc:2454] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 0c370ec2c0a54019a13fb41bc42e60e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5865c945f2284783867b927b15f11b39 in term 0.
I20260925 19:06:30.881850 12303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe2a194a7464453a5f7cd4fa8b3dd83" candidate_uuid: "5865c945f2284783867b927b15f11b39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c370ec2c0a54019a13fb41bc42e60e9"
I20260925 19:06:30.882102 12303 raft_consensus.cc:3037] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 0c370ec2c0a54019a13fb41bc42e60e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:06:30.883028 12303 raft_consensus.cc:2454] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 0c370ec2c0a54019a13fb41bc42e60e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5865c945f2284783867b927b15f11b39 in term 1.
I20260925 19:06:30.883067 12000 catalog_manager.cc:5495] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5865c945f2284783867b927b15f11b39 (127.0.196.195). New cstate: current_term: 1 leader_uuid: "5865c945f2284783867b927b15f11b39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f02e7ecbe8b4b16b640ad065d01f0ee" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5865c945f2284783867b927b15f11b39" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c370ec2c0a54019a13fb41bc42e60e9" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 41451 } health_report { overall_health: UNKNOWN } } }
I20260925 19:06:30.926999 12499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:06:30.924157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58470 (local address 127.0.196.195:35413)
0925 19:06:30.924202 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:06:30.924203 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:06:30.924212 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:06:30.924302 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:06:30.924303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:06:30.924324 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:06:30.924413 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:06:30.924526 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.925101 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.925884 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:06:30.926101 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:06:30.926178 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:06:30.926386 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:06:30.926521 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:06:30.926604 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 19:06:30.926889 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 19:06:30.926915 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:06:30.929497 12303 raft_consensus.cc:1261] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 0c370ec2c0a54019a13fb41bc42e60e9 [term 1 FOLLOWER]: Refusing update from remote peer 5865c945f2284783867b927b15f11b39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:06:30.929497 12148 raft_consensus.cc:1261] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 9f02e7ecbe8b4b16b640ad065d01f0ee [term 1 FOLLOWER]: Refusing update from remote peer 5865c945f2284783867b927b15f11b39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:06:30.929867 12512 consensus_queue.cc:1035] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f02e7ecbe8b4b16b640ad065d01f0ee" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:06:30.929986 12507 consensus_queue.cc:1035] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c370ec2c0a54019a13fb41bc42e60e9" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:06:30.933022 12515 mvcc.cc:205] Tried to move back new op lower bound from 7333327630044319744 to 7333327629851000832. Current Snapshot: MvccSnapshot[applied={T|T < 7333327630044319744}]
I20260925 19:06:30.935299 12516 mvcc.cc:205] Tried to move back new op lower bound from 7333327630044319744 to 7333327629851000832. Current Snapshot: MvccSnapshot[applied={T|T < 7333327630044319744}]
I20260925 19:06:30.936681 12517 mvcc.cc:205] Tried to move back new op lower bound from 7333327630044319744 to 7333327629851000832. Current Snapshot: MvccSnapshot[applied={T|T < 7333327630044319744}]
I20260925 19:06:30.939791 11999 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:06:30.939915 11999 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:06:30.940714 11999 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1fcd20a701c843c69d6d455ab51f9ad3: Sending DeleteTablet for 3 replicas of tablet ebe2a194a7464453a5f7cd4fa8b3dd83
I20260925 19:06:30.941251 12146 tablet_service.cc:1514] Processing DeleteTablet for tablet ebe2a194a7464453a5f7cd4fa8b3dd83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:06:30 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:34702
I20260925 19:06:30.941497 12418 tablet_service.cc:1514] Processing DeleteTablet for tablet ebe2a194a7464453a5f7cd4fa8b3dd83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:06:30 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:58466
I20260925 19:06:30.941973 12284 tablet_service.cc:1514] Processing DeleteTablet for tablet ebe2a194a7464453a5f7cd4fa8b3dd83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:06:30 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:42294
I20260925 19:06:30.943183 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 12081
I20260925 19:06:30.944160 12525 tablet_replica.cc:330] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39: stopping tablet replica
I20260925 19:06:30.944332 12526 tablet_replica.cc:330] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 0c370ec2c0a54019a13fb41bc42e60e9: stopping tablet replica
I20260925 19:06:30.944499 12526 raft_consensus.cc:2229] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 0c370ec2c0a54019a13fb41bc42e60e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:06:30.944618 12525 raft_consensus.cc:2229] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:06:30.944799 12525 raft_consensus.cc:2258] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:06:30.944986 12526 raft_consensus.cc:2258] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 0c370ec2c0a54019a13fb41bc42e60e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:06:30.945729 12525 ts_tablet_manager.cc:1905] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:06:30.947649 12525 ts_tablet_manager.cc:1918] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:06:30.947770 12525 log.cc:1196] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790363087086945-787-0/minicluster-data/ts-2/wal/wals/ebe2a194a7464453a5f7cd4fa8b3dd83
I20260925 19:06:30.948097 12525 ts_tablet_manager.cc:1939] T ebe2a194a7464453a5f7cd4fa8b3dd83 P 5865c945f2284783867b927b15f11b39: Deleting consensus metadata
I20260925 19:06:30.950568 11984 catalog_manager.cc:4841] TS 5865c945f2284783867b927b15f11b39 (127.0.196.195:35413): tablet ebe2a194a7464453a5f7cd4fa8b3dd83 (table test-table [id=d04f516eea2a4cdda6cf2b3ca71332f9]) successfully deleted
W20260925 19:06:30.951030 11984 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.196.193:45015 (error 108)
I20260925 19:06:30.951785 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 12218
I20260925 19:06:30.959741 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 12355
I20260925 19:06:30.961594 12500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:06:30.961456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:45015 (local address 127.0.0.1:34718)
0925 19:06:30.961511 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 19:06:30.961550 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.193:45015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 19:06:30.961829 11984 catalog_manager.cc:4584] TS 9f02e7ecbe8b4b16b640ad065d01f0ee (127.0.196.193:45015): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ebe2a194a7464453a5f7cd4fa8b3dd83: Network error: Client connection negotiation failed: client connection to 127.0.196.193:45015: connect: Connection refused (error 111)
I20260925 19:06:30.962176 12500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:06:30.962101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:41451 (local address 127.0.0.1:42296)
0925 19:06:30.962134 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 19:06:30.962156 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.194:41451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 19:06:30.967109 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 11968
I20260925 19:06:30.973878 787 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 19:06:31.308668 787 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 19:06:31.308811 787 mini_ranger.cc:66] Stopping Ranger...
I20260925 19:06:31.653512 787 mini_ranger.cc:68] Stopped Ranger
2026-09-25 19:06:31.653 UTC [11554] LOG: received smart shutdown request
2026-09-25 19:06:31.657 UTC [11554] LOG: background worker "logical replication launcher" (PID 11563) exited with exit code 1
2026-09-25 19:06:31.657 UTC [11558] LOG: shutting down
2026-09-25 19:06:31.847 UTC [11554] LOG: database system is shut down
2026-09-25T19:06:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48788 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:06:32 dist-test-slave-rn98 krb5kdc[12537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:06:32 dist-test-slave-rn98 krb5kdc[12537](info): set up 2 sockets
Sep 25 19:06:32 dist-test-slave-rn98 krb5kdc[12537](info): commencing operation
krb5kdc: starting...
W20260925 19:06:33.006649 787 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:06:33 dist-test-slave-rn98 krb5kdc[12537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:06:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:06:33Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.196.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.196.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.196.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/rangeradmin_127.0.196.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.196.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/rangeradmin_127.0.196.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.196.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.196.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.196.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/rangerlookup_127.0.196.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.196.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/rangerlookup_127.0.196.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.196.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.196.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.196.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/HTTP_127.0.196.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.196.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/HTTP_127.0.196.212@KRBTEST.COM.keytab.
I20260925 19:06:33.182577 787 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskhGKzmd/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/postgres -l logfile start
2026-09-25 19:06:35.892 UTC [12567] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 19:06:35.893 UTC [12567] LOG: listening on IPv4 address "127.0.196.212", port 41417
2026-09-25 19:06:35.897 UTC [12567] LOG: listening on Unix socket "/tmp/.s.PGSQL.41417"
2026-09-25 19:06:35.933 UTC [12570] LOG: database system was shut down at 2026-09-25 19:06:34 UTC
2026-09-25 19:06:35.938 UTC [12567] LOG: database system is ready to accept connections
I20260925 19:06:36.878438 787 mini_postgres.cc:96] Postgres bound to 41417
2026-09-25 19:06:36.884 UTC [12578] FATAL: database "slave" does not exist
127.0.196.212:41417 - accepting connections
I20260925 19:06:36.885370 787 mini_ranger.cc:161] Starting Ranger...
I20260925 19:06:36.901188 787 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 19:06:37.473627 787 mini_ranger.cc:87] Created ranger Postgres database
I20260925 19:06:37.473747 787 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ranger-admin
2026-09-25 19:06:37,744 [I] DB FLAVOR :POSTGRES
2026-09-25 19:06:37,744 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 19:06:37,744 [I] Checking connection..
2026-09-25 19:06:37,744 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:41417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 19:06:37,985 [I] Checking connection passed.
2026-09-25 19:06:37,985 [I] --------- Verifying version history table ---------
2026-09-25 19:06:37,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:41417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 19:06:38,261 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 19:06:38,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:41417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 19:06:38,506 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 19:06:38,506 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 19:06:38,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:41417/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 19:06:38.765 UTC [12642] WARNING: there is no transaction in progress
2026-09-25 19:06:38,772 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 19:06:38,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:41417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 19:06:39,040 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 19:06:39,041 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 19:06:39,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:41417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 19:06:39,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:41417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 19:06:39,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:41417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rn98.c.gcp-upstream.internal','N') ;"
2026-09-25 19:06:39,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:41417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 19:06:40,072 [I] Table x_portal_user does not exist in database ranger
2026-09-25 19:06:40,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:41417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 19:06:40,311 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 19:06:40,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:41417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 19:06:40,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:41417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 19:06:40,803 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 19:06:40,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:41417/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 19:06:41.872 UTC [12750] WARNING: there is no transaction in progress
2026-09-25 19:06:41.890 UTC [12750] WARNING: there is no transaction in progress
2026-09-25 19:06:41.909 UTC [12750] WARNING: there is no transaction in progress
2026-09-25 19:06:41.926 UTC [12750] WARNING: there is no transaction in progress
2026-09-25 19:06:41.943 UTC [12750] WARNING: there is no transaction in progress
2026-09-25 19:06:41.963 UTC [12750] WARNING: there is no transaction in progress
2026-09-25 19:06:42.024 UTC [12750] WARNING: there is no transaction in progress
2026-09-25 19:06:42.032 UTC [12750] WARNING: there is no transaction in progress
2026-09-25 19:06:42.049 UTC [12750] WARNING: there is no transaction in progress
2026-09-25 19:06:42.061 UTC [12750] WARNING: there is no transaction in progress
2026-09-25 19:06:42.073 UTC [12750] WARNING: there is no transaction in progress
2026-09-25 19:06:42.089 UTC [12750] WARNING: there is no transaction in progress
2026-09-25 19:06:42.102 UTC [12750] WARNING: there is no transaction in progress
2026-09-25 19:06:42.110 UTC [12750] WARNING: there is no transaction in progress
2026-09-25 19:06:42.119 UTC [12750] WARNING: there is no transaction in progress
2026-09-25 19:06:42.127 UTC [12750] WARNING: there is no transaction in progress
2026-09-25 19:06:42,890 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 19:06:42,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:41417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 19:06:43,141 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 19:06:43,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:41417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 19:06:43,383 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 19:06:43,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:41417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 19:06:43,635 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 19:06:43,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:41417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 19:06:43,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:41417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 19:06:44,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:41417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rn98.c.gcp-upstream.internal';"
2026-09-25 19:06:44,373 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 19:06:44,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:41417/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 19:06:44,625 [I] DB_PATCHES have already been applied
I20260925 19:06:44.631162 787 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ranger-admin:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskhGKzmd/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 19:06:44.631229 787 mini_ranger.cc:193] Using host: 127.0.196.212
I20260925 19:06:44.634251 787 mini_ranger.cc:227] Ranger admin URL: http://127.0.196.212:53067
Sep 25, 2026 7:06:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 7:06:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 7:06:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 7:06:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 7:06:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 7:06:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 7:06:45 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 19:06:45.820051 787 mini_ranger.cc:160] Time spent starting Ranger: real 8.935s user 0.000s sys 0.009s
Sep 25, 2026 7:07:06 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 7:07:06 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 7:07:06 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 7:07:07 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 7:07:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 7:07:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 7:07:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 7:07:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 7:07:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 7:07:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 7:07:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 7:07:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 7:07:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 7:07:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 7:07:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 7:07:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 7:07:07 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 7:07:07 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 19:07:08.464870 787 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.196.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.196.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.196.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/rangerkms_127.0.196.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.196.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/rangerkms_127.0.196.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.196.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.196.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.196.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/HTTP_127.0.196.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.196.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/HTTP_127.0.196.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 19:07:08 dist-test-slave-rn98 krb5kdc[12537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363228, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 19:07:08.550451 787 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 19:07:08.560163 787 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 19:07:09.509856 787 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 19:07:09.509948 787 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ranger-kms
I20260925 19:07:09.509961 787 mini_ranger_kms.cc:227] Using postgres at 127.0.196.212:41417
2026-09-25 19:07:09,680 [I] DB FLAVOR :POSTGRES
2026-09-25 19:07:09,680 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 19:07:09,680 [I] Checking connection
2026-09-25 19:07:09,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:41417/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 19:07:09,948 [I] connection success
2026-09-25 19:07:09,948 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 19:07:09,948 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 19:07:09,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:41417/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 19:07:10,209 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 19:07:10,209 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 19:07:10,209 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 19:07:10,210 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.196.212:41417/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 19:07:10,471 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 19:07:10.979684 787 mini_ranger_kms.cc:310] Created kms service
I20260925 19:07:11.150015 787 mini_ranger_kms.cc:326] Created kudu user
I20260925 19:07:11.194692 787 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 19:07:11 dist-test-slave-rn98 krb5kdc[12537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363228, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.196.212@KRBTEST.COM
I20260925 19:07:11.631372 787 mini_ranger_kms.cc:384] Added ranger policy
I20260925 19:07:11.631492 787 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ranger-kms:/tmp/dist-test-taskhGKzmd/build/release/bin/postgresql.jar:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskhGKzmd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 19:07:11.631510 787 mini_ranger_kms.cc:240] Using host: 127.0.196.212
I20260925 19:07:11.635485 787 mini_ranger_kms.cc:276] Ranger KMS PID: 12931
I20260925 19:07:11.635579 787 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.196.212:46595
Sep 25, 2026 7:07:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 7:07:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 7:07:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 7:07:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 7:07:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 7:07:13 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 19:07:13.886422 787 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.336s user 0.000s sys 0.149s
I20260925 19:07:13.886520 787 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 19:07:13.886567 787 mini_ranger_kms.cc:401] 127.0.196.212:46595/kms/v1/keys
Sep 25 19:07:14 dist-test-slave-rn98 krb5kdc[12537](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790363234, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.196.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 19:07:14 dist-test-slave-rn98 krb5kdc[12537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363234, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.196.212@KRBTEST.COM for HTTP/127.0.196.212@KRBTEST.COM
Sep 25 19:07:14 dist-test-slave-rn98 krb5kdc[12537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363234, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.196.212@KRBTEST.COM for HTTP/127.0.196.212@KRBTEST.COM
Sep 25 19:07:17 dist-test-slave-rn98 krb5kdc[12537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:07:17.250290 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:39661
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:39661
--ranger_config_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.196.212:46595/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:07:17.352149 12981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:07:17.352372 12981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:07:17.352396 12981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:07:17.353629 12981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:07:17.353677 12981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:07:17.353690 12981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:07:17.353703 12981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:07:17.353715 12981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:07:17.355342 12981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.196.212:46595/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:39661
--ranger_config_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:39661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:07:17.355540 12981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:07:17.355825 12981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:07:17.360813 12989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:17.360839 12986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:17.360790 12987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:17.360844 12981 server_base.cc:1034] running on GCE node
I20260925 19:07:17.361248 12981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:07:17.361502 12981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:07:17.362677 12981 hybrid_clock.cc:648] HybridClock initialized: now 1790363237362657 us; error 39 us; skew 500 ppm
Sep 25 19:07:17 dist-test-slave-rn98 krb5kdc[12537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:17.365236 12981 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:07:17.365706 12981 webserver.cc:460] Webserver started at http://127.0.196.254:40837/ using document root <none> and password file <none>
I20260925 19:07:17.365943 12981 fs_manager.cc:359] Metadata directory not provided
I20260925 19:07:17.366000 12981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:07:17.366155 12981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 19:07:17 dist-test-slave-rn98 krb5kdc[12537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for HTTP/127.0.196.212@KRBTEST.COM
I20260925 19:07:17.412607 12981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "5399492cbf3d4c75ac32f1d05c70f2fe"
format_stamp: "Formatted at 2026-09-25 19:07:17 on dist-test-slave-rn98"
server_key: "326e82d92955bea5caf4b0f4afcbc8cb"
server_key_iv: "0346266ab16ee98761045b8e0134ed6c"
server_key_version: "kuduclusterkey@0"
I20260925 19:07:17.413012 12981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "5399492cbf3d4c75ac32f1d05c70f2fe"
format_stamp: "Formatted at 2026-09-25 19:07:17 on dist-test-slave-rn98"
server_key: "326e82d92955bea5caf4b0f4afcbc8cb"
server_key_iv: "0346266ab16ee98761045b8e0134ed6c"
server_key_version: "kuduclusterkey@0"
I20260925 19:07:17.414608 12981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 19:07:17.431916 12996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:17.432158 12981 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:07:17.432261 12981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "5399492cbf3d4c75ac32f1d05c70f2fe"
format_stamp: "Formatted at 2026-09-25 19:07:17 on dist-test-slave-rn98"
server_key: "326e82d92955bea5caf4b0f4afcbc8cb"
server_key_iv: "0346266ab16ee98761045b8e0134ed6c"
server_key_version: "kuduclusterkey@0"
I20260925 19:07:17.432363 12981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:07:17.468098 12981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:07:17.470660 12981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:07:17.470831 12981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:07:17.476148 12981 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:39661
I20260925 19:07:17.476209 13048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:39661 every 8 connection(s)
I20260925 19:07:17.476630 12981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:07:17.477347 13049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:07:17.478157 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 12981
I20260925 19:07:17.478263 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:07:17.478443 787 external_mini_cluster.cc:1428] Setting key 1844a8f3037f948fe0de9ade85e1e2e1
I20260925 19:07:17.481312 13049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe: Bootstrap starting.
I20260925 19:07:17.482051 13049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe: Neither blocks nor log segments found. Creating new log.
I20260925 19:07:17.482348 13049 log.cc:824] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe: Log is configured to *not* fsync() on all Append() calls
I20260925 19:07:17.483168 13049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe: No bootstrap required, opened a new log
I20260925 19:07:17.486012 13049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5399492cbf3d4c75ac32f1d05c70f2fe" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 39661 } }
I20260925 19:07:17.486203 13049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:07:17.486250 13049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5399492cbf3d4c75ac32f1d05c70f2fe, State: Initialized, Role: FOLLOWER
I20260925 19:07:17.486377 13049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5399492cbf3d4c75ac32f1d05c70f2fe" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 39661 } }
I20260925 19:07:17.486464 13049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:07:17.486500 13049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:07:17.486541 13049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [term 0 FOLLOWER]: Advancing to term 1
Sep 25 19:07:17 dist-test-slave-rn98 krb5kdc[12537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:07:17.487401 13049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5399492cbf3d4c75ac32f1d05c70f2fe" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 39661 } }
I20260925 19:07:17.487514 13049 leader_election.cc:302] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5399492cbf3d4c75ac32f1d05c70f2fe; no voters:
I20260925 19:07:17.487718 13049 leader_election.cc:288] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:07:17.487769 13054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:07:17.487936 13054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [term 1 LEADER]: Becoming Leader. State: Replica: 5399492cbf3d4c75ac32f1d05c70f2fe, State: Running, Role: LEADER
I20260925 19:07:17.488034 13049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:07:17.488034 13054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5399492cbf3d4c75ac32f1d05c70f2fe" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 39661 } }
I20260925 19:07:17.488437 13054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5399492cbf3d4c75ac32f1d05c70f2fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5399492cbf3d4c75ac32f1d05c70f2fe" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 39661 } } }
I20260925 19:07:17.488513 13054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [sys.catalog]: This master's current role is: LEADER
I20260925 19:07:17.488652 13054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5399492cbf3d4c75ac32f1d05c70f2fe. Latest consensus state: current_term: 1 leader_uuid: "5399492cbf3d4c75ac32f1d05c70f2fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5399492cbf3d4c75ac32f1d05c70f2fe" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 39661 } } }
I20260925 19:07:17.488708 13054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [sys.catalog]: This master's current role is: LEADER
I20260925 19:07:17.489750 13049 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 19:07:17.493738 13060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:07:17.496384 13060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:07:17.496570 13052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:17.479007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49520 (local address 127.0.196.254:39661)
0925 19:07:17.479214 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:17.479225 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:17.479258 (+ 33us) server_negotiation.cc:407] Connection header received
0925 19:07:17.479830 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:17.479840 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:17.479908 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:17.480064 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:17.480939 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:17.482533 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:17.483732 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:17.484409 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:17.484540 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:17.488445 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:17.488450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:17.488450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:17.488463 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:17.493259 (+ 4796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:17.493851 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:17.493856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:17.493856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:17.494002 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:17.494329 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:17.494330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:17.494330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:17.495512 (+ 1182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:17.495732 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:17.496008 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 19:07:17.496209 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":82,"threads_started":1}
I20260925 19:07:17.498399 13060 catalog_manager.cc:1347] Generated new cluster ID: 04403298ea164ced9490f7380265fba2
I20260925 19:07:17.498450 13060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:07:17.522387 13060 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:07:17.523267 13060 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:07:17.528620 13060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe: Generated new TSK 0
I20260925 19:07:17.528937 13060 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
I20260925 19:07:18.298954 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.196.212:46595/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:0
--local_ip_for_outbound_sockets=127.0.196.193
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:39661
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:07:18.414920 13094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:07:18.415153 13094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:07:18.415194 13094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:07:18.416355 13094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:07:18.416424 13094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:07:18.416468 13094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:07:18.417970 13094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.196.212:46595/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:39661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:07:18.418239 13094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:07:18.418470 13094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:07:18.423754 13094 server_base.cc:1034] running on GCE node
W20260925 19:07:18.423696 13099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:18.423709 13102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:18.423882 13100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:18.424247 13094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:07:18.424506 13094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:07:18.425665 13094 hybrid_clock.cc:648] HybridClock initialized: now 1790363238425648 us; error 34 us; skew 500 ppm
Sep 25 19:07:18 dist-test-slave-rn98 krb5kdc[12537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:18.428831 13094 init.cc:377] Logged in from keytab as kudu/127.0.196.193@KRBTEST.COM (short username kudu)
I20260925 19:07:18.429381 13094 webserver.cc:460] Webserver started at http://127.0.196.193:41949/ using document root <none> and password file <none>
I20260925 19:07:18.429615 13094 fs_manager.cc:359] Metadata directory not provided
I20260925 19:07:18.429682 13094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:07:18.429842 13094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 19:07:18 dist-test-slave-rn98 krb5kdc[12537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for HTTP/127.0.196.212@KRBTEST.COM
I20260925 19:07:18.445340 13094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/data/instance:
uuid: "c8c1e1a7bc7849d396b55a3bdab11c09"
format_stamp: "Formatted at 2026-09-25 19:07:18 on dist-test-slave-rn98"
server_key: "ae5be703c05ac0c8bd975f7bbd947655"
server_key_iv: "efbc9f63f607875464bf246e6907f0dd"
server_key_version: "kuduclusterkey@0"
I20260925 19:07:18.446185 13094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/wal/instance:
uuid: "c8c1e1a7bc7849d396b55a3bdab11c09"
format_stamp: "Formatted at 2026-09-25 19:07:18 on dist-test-slave-rn98"
server_key: "ae5be703c05ac0c8bd975f7bbd947655"
server_key_iv: "efbc9f63f607875464bf246e6907f0dd"
server_key_version: "kuduclusterkey@0"
I20260925 19:07:18.448195 13094 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 19:07:18.463493 13109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:18.463778 13094 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:07:18.463903 13094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "c8c1e1a7bc7849d396b55a3bdab11c09"
format_stamp: "Formatted at 2026-09-25 19:07:18 on dist-test-slave-rn98"
server_key: "ae5be703c05ac0c8bd975f7bbd947655"
server_key_iv: "efbc9f63f607875464bf246e6907f0dd"
server_key_version: "kuduclusterkey@0"
I20260925 19:07:18.464016 13094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:07:18.495446 13094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:07:18.498149 13094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:07:18.498306 13094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:07:18.498584 13094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:07:18.499094 13094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:07:18.499176 13094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:18.499210 13094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:07:18.499264 13094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:18.506793 13094 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:43789
I20260925 19:07:18.507253 13094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
I20260925 19:07:18.507757 13224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:43789 every 8 connection(s)
I20260925 19:07:18.511092 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 13094
I20260925 19:07:18.511204 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/wal/instance
I20260925 19:07:18.511397 787 external_mini_cluster.cc:1428] Setting key 8471cd29ea70eae297bd755197be5c7f
Sep 25 19:07:18 dist-test-slave-rn98 krb5kdc[12537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:07:18.520661 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:18.508374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:39661 (local address 127.0.196.193:60375)
0925 19:07:18.508577 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:18.508629 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:18.509336 (+ 707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:18.509768 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:18.509772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:18.510040 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:07:18.510350 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:18.510357 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:18.511431 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:18.511432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:18.513354 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:18.513358 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:18.513932 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:18.513933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:18.514094 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:18.514790 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:07:18.514801 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:07:18.516566 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:07:18.518807 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:18.518817 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:18.518820 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:18.519182 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:18.519417 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:18.519418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:18.519419 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:18.519568 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:18.519875 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:07:18.519881 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:07:18.520086 (+ 205us) client_negotiation.cc:769] Sending connection context
0925 19:07:18.520213 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 19:07:18.520397 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for kudu/127.0.196.194@KRBTEST.COM; defaulting to no policy
I20260925 19:07:18.521466 13228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:18.508428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:60375 (local address 127.0.196.254:39661)
0925 19:07:18.508580 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:18.508582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:18.508728 (+ 146us) server_negotiation.cc:407] Connection header received
0925 19:07:18.509530 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:18.509531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:18.509546 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:18.509621 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:18.510471 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:18.511315 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:18.513461 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:18.513767 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:18.513843 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:18.516690 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:18.516692 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:18.516692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:18.516701 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:18.518698 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:18.519283 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:18.519284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:18.519285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:18.519352 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:18.519636 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:18.519637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:18.519637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:18.519790 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:18.519848 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:18.521261 (+ 1413us) server_negotiation.cc:299] Negotiation successful
0925 19:07:18.521361 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
I20260925 19:07:18.522127 13225 heartbeater.cc:347] Connected to a master server at 127.0.196.254:39661
I20260925 19:07:18.522236 13225 heartbeater.cc:464] Registering TS with master...
I20260925 19:07:18.522449 13225 heartbeater.cc:511] Master 127.0.196.254:39661 requested a full tablet report, sending...
I20260925 19:07:18.523129 13013 ts_manager.cc:194] Registered new tserver with Master: c8c1e1a7bc7849d396b55a3bdab11c09 (127.0.196.193:43789)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.194@KRBTEST.COM" created.
I20260925 19:07:18.524107 13013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.193@KRBTEST.COM'} at 127.0.196.193:60375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
I20260925 19:07:18.539867 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.196.212:46595/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:0
--local_ip_for_outbound_sockets=127.0.196.194
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:39661
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:07:18.646836 13231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:07:18.647094 13231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:07:18.647135 13231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:07:18.648723 13231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:07:18.648809 13231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:07:18.648859 13231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:07:18.650699 13231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.196.212:46595/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:39661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:07:18.650985 13231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:07:18.651278 13231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:07:18.656883 13241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:18.656822 13239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:18.656818 13238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:18.657604 13231 server_base.cc:1034] running on GCE node
I20260925 19:07:18.657855 13231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:07:18.658128 13231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:07:18.659281 13231 hybrid_clock.cc:648] HybridClock initialized: now 1790363238659261 us; error 36 us; skew 500 ppm
Sep 25 19:07:18 dist-test-slave-rn98 krb5kdc[12537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:18.662106 13231 init.cc:377] Logged in from keytab as kudu/127.0.196.194@KRBTEST.COM (short username kudu)
I20260925 19:07:18.662585 13231 webserver.cc:460] Webserver started at http://127.0.196.194:46511/ using document root <none> and password file <none>
I20260925 19:07:18.662850 13231 fs_manager.cc:359] Metadata directory not provided
I20260925 19:07:18.662923 13231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:07:18.663071 13231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 19:07:18 dist-test-slave-rn98 krb5kdc[12537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for HTTP/127.0.196.212@KRBTEST.COM
Sep 25 19:07:18 dist-test-slave-rn98 krb5kdc[12537](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790363238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:18.677515 13231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/data/instance:
uuid: "338113a4b2014022ae4ff0ee8882217a"
format_stamp: "Formatted at 2026-09-25 19:07:18 on dist-test-slave-rn98"
server_key: "669c9574c08c3c8349769a6791b4f3e7"
server_key_iv: "2309a2563feb34b59ae7949cb3f928ea"
server_key_version: "kuduclusterkey@0"
I20260925 19:07:18.677950 13231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/wal/instance:
uuid: "338113a4b2014022ae4ff0ee8882217a"
format_stamp: "Formatted at 2026-09-25 19:07:18 on dist-test-slave-rn98"
server_key: "669c9574c08c3c8349769a6791b4f3e7"
server_key_iv: "2309a2563feb34b59ae7949cb3f928ea"
server_key_version: "kuduclusterkey@0"
I20260925 19:07:18.680665 13231 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260925 19:07:18.713893 13248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:18.714118 13231 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.001s sys 0.000s
I20260925 19:07:18.714219 13231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "338113a4b2014022ae4ff0ee8882217a"
format_stamp: "Formatted at 2026-09-25 19:07:18 on dist-test-slave-rn98"
server_key: "669c9574c08c3c8349769a6791b4f3e7"
server_key_iv: "2309a2563feb34b59ae7949cb3f928ea"
server_key_version: "kuduclusterkey@0"
I20260925 19:07:18.714349 13231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:07:18.728161 13231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:07:18.730795 13231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:07:18.731006 13231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:07:18.731324 13231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:07:18.731792 13231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:07:18.731882 13231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:18.731951 13231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:07:18.732009 13231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:18.738686 13231 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:41761
I20260925 19:07:18.739195 13231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
I20260925 19:07:18.742715 13361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:41761 every 8 connection(s)
I20260925 19:07:18.748517 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 13231
I20260925 19:07:18.748620 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/wal/instance
I20260925 19:07:18.748824 787 external_mini_cluster.cc:1428] Setting key 4cb6bf5eeaa616a9635cb04dbb9ed9cd
Sep 25 19:07:18 dist-test-slave-rn98 krb5kdc[12537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:07:18.757189 13228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:18.742777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:41231 (local address 127.0.196.254:39661)
0925 19:07:18.742830 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:18.742832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:18.743401 (+ 569us) server_negotiation.cc:407] Connection header received
0925 19:07:18.744158 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:18.744159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:18.744175 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:18.744264 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:18.745033 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:18.745855 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:18.747486 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:18.747815 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:18.747896 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:18.753661 (+ 5765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:18.753662 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:18.753663 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:18.753671 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:18.755495 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:18.756031 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:18.756032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:18.756033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:18.756091 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:18.756396 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:18.756397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:18.756398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:18.756540 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:18.756585 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:18.756994 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 19:07:18.757092 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:07:18.758213 13365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:18.742960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:39661 (local address 127.0.196.194:41231)
0925 19:07:18.743267 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:18.743312 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:18.743981 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:18.744363 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:18.744368 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:18.744627 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:07:18.744914 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:18.744921 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:18.745971 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:18.745972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:18.747381 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:18.747384 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:18.747935 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:18.747936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:18.748088 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:18.751822 (+ 3734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:07:18.751833 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:07:18.753531 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:07:18.755609 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:18.755618 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:18.755621 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:18.755937 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:18.756161 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:18.756161 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:18.756162 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:18.756319 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:18.756636 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:07:18.756642 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:07:18.756825 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 19:07:18.757538 (+ 713us) client_negotiation.cc:240] Negotiation successful
0925 19:07:18.757724 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":116,"threads_started":1}
I20260925 19:07:18.759042 13362 heartbeater.cc:347] Connected to a master server at 127.0.196.254:39661
I20260925 19:07:18.759212 13362 heartbeater.cc:464] Registering TS with master...
I20260925 19:07:18.759464 13362 heartbeater.cc:511] Master 127.0.196.254:39661 requested a full tablet report, sending...
I20260925 19:07:18.759865 13013 ts_manager.cc:194] Registered new tserver with Master: 338113a4b2014022ae4ff0ee8882217a (127.0.196.194:41761)
I20260925 19:07:18.760366 13013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.194@KRBTEST.COM'} at 127.0.196.194:41231
WARNING: no policy specified for kudu/127.0.196.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
I20260925 19:07:18.789625 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.196.212:46595/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:0
--local_ip_for_outbound_sockets=127.0.196.195
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:39661
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:07:18.912623 13368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:07:18.912948 13368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:07:18.913007 13368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:07:18.914875 13368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:07:18.914942 13368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:07:18.914984 13368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:07:18.916666 13368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.196.212:46595/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:39661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:07:18.916976 13368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:07:18.917227 13368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:07:18.922747 13374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:18.922775 13373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:18.922863 13376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:18.922887 13368 server_base.cc:1034] running on GCE node
I20260925 19:07:18.923187 13368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:07:18.923442 13368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:07:18.924604 13368 hybrid_clock.cc:648] HybridClock initialized: now 1790363238924583 us; error 40 us; skew 500 ppm
Sep 25 19:07:18 dist-test-slave-rn98 krb5kdc[12537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:18.927250 13368 init.cc:377] Logged in from keytab as kudu/127.0.196.195@KRBTEST.COM (short username kudu)
I20260925 19:07:18.927829 13368 webserver.cc:460] Webserver started at http://127.0.196.195:46013/ using document root <none> and password file <none>
I20260925 19:07:18.928083 13368 fs_manager.cc:359] Metadata directory not provided
I20260925 19:07:18.928150 13368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:07:18.928303 13368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 19:07:18 dist-test-slave-rn98 krb5kdc[12537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for HTTP/127.0.196.212@KRBTEST.COM
I20260925 19:07:18.941061 13368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/data/instance:
uuid: "9fd3dfc7689746a39dfdd7f1de906b88"
format_stamp: "Formatted at 2026-09-25 19:07:18 on dist-test-slave-rn98"
server_key: "6c1308722d5dc9ca940a6946f5eedecd"
server_key_iv: "1d8d0ba5b83df941c7ec742bc2b01135"
server_key_version: "kuduclusterkey@0"
I20260925 19:07:18.941541 13368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/wal/instance:
uuid: "9fd3dfc7689746a39dfdd7f1de906b88"
format_stamp: "Formatted at 2026-09-25 19:07:18 on dist-test-slave-rn98"
server_key: "6c1308722d5dc9ca940a6946f5eedecd"
server_key_iv: "1d8d0ba5b83df941c7ec742bc2b01135"
server_key_version: "kuduclusterkey@0"
I20260925 19:07:18.942967 13368 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260925 19:07:18.964922 13383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:18.965284 13368 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.001s
I20260925 19:07:18.965406 13368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "9fd3dfc7689746a39dfdd7f1de906b88"
format_stamp: "Formatted at 2026-09-25 19:07:18 on dist-test-slave-rn98"
server_key: "6c1308722d5dc9ca940a6946f5eedecd"
server_key_iv: "1d8d0ba5b83df941c7ec742bc2b01135"
server_key_version: "kuduclusterkey@0"
I20260925 19:07:18.965521 13368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:07:18.989198 13368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:07:18.992017 13368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:07:18.992187 13368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:07:18.992467 13368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:07:18.992846 13368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:07:18.992900 13368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:18.992939 13368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:07:18.992965 13368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:19.000126 13368 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:38965
I20260925 19:07:19.000275 13496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:38965 every 8 connection(s)
I20260925 19:07:19.000556 13368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
Sep 25 19:07:19 dist-test-slave-rn98 krb5kdc[12537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:07:19.010716 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 13368
I20260925 19:07:19.010815 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/wal/instance
I20260925 19:07:19.011040 787 external_mini_cluster.cc:1428] Setting key 463922580777e3e0be20436cdfc4f4e7
I20260925 19:07:19.011202 13228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:19.001581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:47733 (local address 127.0.196.254:39661)
0925 19:07:19.001629 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:19.001632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:19.001870 (+ 238us) server_negotiation.cc:407] Connection header received
0925 19:07:19.002424 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:19.002425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:19.002439 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:19.002485 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:19.003146 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:19.003711 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:19.005179 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:19.005476 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:19.005525 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:19.008074 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:19.008075 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:19.008076 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:19.008084 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:19.009586 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:19.010022 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:19.010024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:19.010024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:19.010080 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:19.010354 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:19.010355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:19.010356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:19.010499 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:19.010564 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:19.011004 (+ 440us) server_negotiation.cc:299] Negotiation successful
0925 19:07:19.011106 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:07:19.011771 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:19.001588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:39661 (local address 127.0.196.195:47733)
0925 19:07:19.001751 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:19.001793 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:19.002303 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:19.002564 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:19.002569 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:19.002842 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:07:19.003048 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:19.003052 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:19.003825 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:19.003826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:19.005072 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:19.005075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:19.005643 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:19.005644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:19.005787 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:19.006455 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:07:19.006465 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:07:19.007970 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:07:19.009691 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:19.009698 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:19.009701 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:19.009929 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:19.010156 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:19.010157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:19.010157 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:19.010278 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:19.010594 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:07:19.010601 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:07:19.010827 (+ 226us) client_negotiation.cc:769] Sending connection context
0925 19:07:19.011336 (+ 509us) client_negotiation.cc:240] Negotiation successful
0925 19:07:19.011517 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260925 19:07:19.012404 13497 heartbeater.cc:347] Connected to a master server at 127.0.196.254:39661
I20260925 19:07:19.012503 13497 heartbeater.cc:464] Registering TS with master...
I20260925 19:07:19.012691 13497 heartbeater.cc:511] Master 127.0.196.254:39661 requested a full tablet report, sending...
I20260925 19:07:19.013058 13013 ts_manager.cc:194] Registered new tserver with Master: 9fd3dfc7689746a39dfdd7f1de906b88 (127.0.196.195:38965)
I20260925 19:07:19.013548 13013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.195@KRBTEST.COM'} at 127.0.196.195:47733
I20260925 19:07:19.015058 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:07:19.015676 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 13094
I20260925 19:07:19.021693 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 13231
I20260925 19:07:19.027328 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 13368
I20260925 19:07:19.033213 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 12981
I20260925 19:07:19.038894 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:39661
--webserver_interface=127.0.196.254
--webserver_port=40837
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:39661
--ranger_config_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.196.212:46595/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.196.212:46595/kms with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:07:19.121559 13500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:07:19.121781 13500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:07:19.121803 13500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:07:19.122931 13500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:07:19.122970 13500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:07:19.122985 13500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:07:19.123041 13500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:07:19.123066 13500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:07:19.124666 13500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.196.212:46595/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:39661
--ranger_config_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:39661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=40837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:07:19.124866 13500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:07:19.125032 13500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:07:19.129935 13508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:19.129935 13505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:19.129930 13506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:19.130189 13500 server_base.cc:1034] running on GCE node
I20260925 19:07:19.130427 13500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:07:19.130743 13500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:07:19.131888 13500 hybrid_clock.cc:648] HybridClock initialized: now 1790363239131872 us; error 34 us; skew 500 ppm
Sep 25 19:07:19 dist-test-slave-rn98 krb5kdc[12537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:19.134310 13500 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:07:19.134824 13500 webserver.cc:460] Webserver started at http://127.0.196.254:40837/ using document root <none> and password file <none>
I20260925 19:07:19.135044 13500 fs_manager.cc:359] Metadata directory not provided
I20260925 19:07:19.135114 13500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 19:07:19 dist-test-slave-rn98 krb5kdc[12537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for HTTP/127.0.196.212@KRBTEST.COM
I20260925 19:07:19.196967 13500 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:07:19.197824 13516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:19.198045 13500 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:07:19.198177 13500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "5399492cbf3d4c75ac32f1d05c70f2fe"
format_stamp: "Formatted at 2026-09-25 19:07:17 on dist-test-slave-rn98"
server_key: "326e82d92955bea5caf4b0f4afcbc8cb"
server_key_iv: "0346266ab16ee98761045b8e0134ed6c"
server_key_version: "kuduclusterkey@0"
I20260925 19:07:19.198616 13500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:07:19.213387 13500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:07:19.215547 13500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:07:19.215714 13500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:07:19.220608 13500 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:39661
I20260925 19:07:19.220662 13568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:39661 every 8 connection(s)
I20260925 19:07:19.221064 13500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:07:19.222285 13569 sys_catalog.cc:263] Verifying existing consensus state
I20260925 19:07:19.223023 13569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe: Bootstrap starting.
I20260925 19:07:19.224615 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 13500
I20260925 19:07:19.224938 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.196.212:46595/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:43789
--local_ip_for_outbound_sockets=127.0.196.193
--tserver_master_addrs=127.0.196.254:39661
--webserver_port=41949
--webserver_interface=127.0.196.193
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.196.212:46595/kms with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:07:19.231256 13569 log.cc:824] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe: Log is configured to *not* fsync() on all Append() calls
I20260925 19:07:19.233039 13569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:07:19.233341 13569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe: Bootstrap complete.
I20260925 19:07:19.235841 13569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5399492cbf3d4c75ac32f1d05c70f2fe" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 39661 } }
I20260925 19:07:19.236106 13569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5399492cbf3d4c75ac32f1d05c70f2fe, State: Initialized, Role: FOLLOWER
I20260925 19:07:19.236315 13569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5399492cbf3d4c75ac32f1d05c70f2fe" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 39661 } }
I20260925 19:07:19.236410 13569 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:07:19.236464 13569 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:07:19.236512 13569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [term 1 FOLLOWER]: Advancing to term 2
I20260925 19:07:19.237521 13569 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5399492cbf3d4c75ac32f1d05c70f2fe" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 39661 } }
I20260925 19:07:19.237643 13569 leader_election.cc:302] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5399492cbf3d4c75ac32f1d05c70f2fe; no voters:
I20260925 19:07:19.237854 13569 leader_election.cc:288] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 19:07:19.237893 13573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [term 2 FOLLOWER]: Leader election won for term 2
I20260925 19:07:19.238001 13573 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [term 2 LEADER]: Becoming Leader. State: Replica: 5399492cbf3d4c75ac32f1d05c70f2fe, State: Running, Role: LEADER
I20260925 19:07:19.238087 13573 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5399492cbf3d4c75ac32f1d05c70f2fe" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 39661 } }
I20260925 19:07:19.238130 13569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:07:19.238377 13574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5399492cbf3d4c75ac32f1d05c70f2fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5399492cbf3d4c75ac32f1d05c70f2fe" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 39661 } } }
I20260925 19:07:19.238464 13574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [sys.catalog]: This master's current role is: LEADER
I20260925 19:07:19.238370 13573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5399492cbf3d4c75ac32f1d05c70f2fe. Latest consensus state: current_term: 2 leader_uuid: "5399492cbf3d4c75ac32f1d05c70f2fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5399492cbf3d4c75ac32f1d05c70f2fe" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 39661 } } }
I20260925 19:07:19.238675 13573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe [sys.catalog]: This master's current role is: LEADER
I20260925 19:07:19.239131 13577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:07:19.239681 13577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:07:19.239814 13577 catalog_manager.cc:1259] Loaded cluster ID: 04403298ea164ced9490f7380265fba2
I20260925 19:07:19.239851 13577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:07:19.240036 13569 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 19:07:19.243434 13577 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:07:19.244030 13577 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe: Loaded TSK: 0
I20260925 19:07:19.244338 13577 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 19:07:19.323733 13571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:07:19.323989 13571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:07:19.324030 13571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:07:19.325361 13571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:07:19.325435 13571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:07:19.325484 13571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:07:19.327387 13571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.196.212:46595/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:43789
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=41949
--tserver_master_addrs=127.0.196.254:39661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:07:19.327672 13571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:07:19.327966 13571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:07:19.333945 13609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:19.334187 13606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:19.334597 13607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:19.335366 13571 server_base.cc:1034] running on GCE node
I20260925 19:07:19.335589 13571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:07:19.335896 13571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:07:19.339210 13571 hybrid_clock.cc:648] HybridClock initialized: now 1790363239339163 us; error 29 us; skew 500 ppm
Sep 25 19:07:19 dist-test-slave-rn98 krb5kdc[12537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:19.342201 13571 init.cc:377] Logged in from keytab as kudu/127.0.196.193@KRBTEST.COM (short username kudu)
I20260925 19:07:19.342895 13571 webserver.cc:460] Webserver started at http://127.0.196.193:41949/ using document root <none> and password file <none>
I20260925 19:07:19.344877 13571 fs_manager.cc:359] Metadata directory not provided
I20260925 19:07:19.345036 13571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 19:07:19 dist-test-slave-rn98 krb5kdc[12537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for HTTP/127.0.196.212@KRBTEST.COM
I20260925 19:07:19.363859 13571 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 19:07:19.364718 13617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:19.364948 13571 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:07:19.365046 13571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "c8c1e1a7bc7849d396b55a3bdab11c09"
format_stamp: "Formatted at 2026-09-25 19:07:18 on dist-test-slave-rn98"
server_key: "ae5be703c05ac0c8bd975f7bbd947655"
server_key_iv: "efbc9f63f607875464bf246e6907f0dd"
server_key_version: "kuduclusterkey@0"
I20260925 19:07:19.365445 13571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:07:19.377555 13571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:07:19.380218 13571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:07:19.380473 13571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:07:19.380774 13571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:07:19.381211 13571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:07:19.381312 13571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:19.381378 13571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:07:19.381413 13571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:19.388144 13571 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:43789
I20260925 19:07:19.388772 13571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
I20260925 19:07:19.389680 13730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:43789 every 8 connection(s)
I20260925 19:07:19.390831 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 13571
I20260925 19:07:19.391130 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.196.212:46595/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:41761
--local_ip_for_outbound_sockets=127.0.196.194
--tserver_master_addrs=127.0.196.254:39661
--webserver_port=46511
--webserver_interface=127.0.196.194
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.196.212:46595/kms with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 19:07:19 dist-test-slave-rn98 krb5kdc[12537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:07:19.404781 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:19.389983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:57019 (local address 127.0.196.254:39661)
0925 19:07:19.390197 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:19.390208 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:19.390429 (+ 221us) server_negotiation.cc:407] Connection header received
0925 19:07:19.391381 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:19.391388 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:19.391449 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:19.391585 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:19.392491 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:19.393693 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:19.395361 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:19.396015 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:19.396106 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:19.400569 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:19.400575 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:19.400575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:19.400595 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:19.402862 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:19.403345 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:19.403348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:19.403349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:19.403450 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:19.403740 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:19.403741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:19.403742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:19.403999 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:19.404089 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:19.404423 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 19:07:19.404533 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":116,"threads_started":1}
I20260925 19:07:19.404781 13733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:19.389565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:39661 (local address 127.0.196.193:57019)
0925 19:07:19.390265 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:19.390311 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:19.391065 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:19.391780 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:19.391784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:19.392052 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:07:19.392341 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:19.392347 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:19.393824 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:19.393825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:19.395234 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:19.395238 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:19.396256 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:19.396258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:19.396405 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:19.397212 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:07:19.397224 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:07:19.400051 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:07:19.402964 (+ 2913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:19.402973 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:19.402975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:19.403239 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:19.403535 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:19.403536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:19.403536 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:19.403662 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:19.404081 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:07:19.404088 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:07:19.404233 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 19:07:19.404341 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 19:07:19.404524 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":84,"threads_started":1}
I20260925 19:07:19.406136 13731 heartbeater.cc:347] Connected to a master server at 127.0.196.254:39661
I20260925 19:07:19.406261 13731 heartbeater.cc:464] Registering TS with master...
I20260925 19:07:19.406458 13731 heartbeater.cc:511] Master 127.0.196.254:39661 requested a full tablet report, sending...
W20260925 19:07:19.407305 13731 heartbeater.cc:644] Failed to heartbeat to 127.0.196.254:39661 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 19:07:19.407390 13731 heartbeater.cc:464] Registering TS with master...
I20260925 19:07:19.407461 13731 heartbeater.cc:511] Master 127.0.196.254:39661 requested a full tablet report, sending...
I20260925 19:07:19.407781 13731 heartbeater.cc:464] Registering TS with master...
I20260925 19:07:19.407857 13731 heartbeater.cc:511] Master 127.0.196.254:39661 requested a full tablet report, sending...
W20260925 19:07:19.408121 13731 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 19:07:19.508699 13735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:07:19.509097 13735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:07:19.509186 13735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:07:19.511054 13735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:07:19.511170 13735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:07:19.511252 13735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:07:19.513477 13735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.196.212:46595/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:41761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=46511
--tserver_master_addrs=127.0.196.254:39661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:07:19.513978 13735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:07:19.514261 13735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:07:19.519941 13743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:19.520196 13740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:19.520447 13741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:19.520565 13735 server_base.cc:1034] running on GCE node
I20260925 19:07:19.520869 13735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:07:19.521162 13735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:07:19.522315 13735 hybrid_clock.cc:648] HybridClock initialized: now 1790363239522286 us; error 55 us; skew 500 ppm
Sep 25 19:07:19 dist-test-slave-rn98 krb5kdc[12537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:19.525059 13735 init.cc:377] Logged in from keytab as kudu/127.0.196.194@KRBTEST.COM (short username kudu)
I20260925 19:07:19.525650 13735 webserver.cc:460] Webserver started at http://127.0.196.194:46511/ using document root <none> and password file <none>
I20260925 19:07:19.525910 13735 fs_manager.cc:359] Metadata directory not provided
I20260925 19:07:19.525969 13735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 19:07:19 dist-test-slave-rn98 krb5kdc[12537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for HTTP/127.0.196.212@KRBTEST.COM
I20260925 19:07:19.543566 13735 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 19:07:19.544581 13751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:19.544834 13735 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:07:19.544986 13735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "338113a4b2014022ae4ff0ee8882217a"
format_stamp: "Formatted at 2026-09-25 19:07:18 on dist-test-slave-rn98"
server_key: "669c9574c08c3c8349769a6791b4f3e7"
server_key_iv: "2309a2563feb34b59ae7949cb3f928ea"
server_key_version: "kuduclusterkey@0"
I20260925 19:07:19.545373 13735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:07:19.567607 13735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:07:19.570251 13735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:07:19.570458 13735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:07:19.570768 13735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:07:19.571167 13735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:07:19.571250 13735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:19.571310 13735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:07:19.571352 13735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:19.578964 13735 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:41761
I20260925 19:07:19.579561 13735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
I20260925 19:07:19.580305 13864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:41761 every 8 connection(s)
I20260925 19:07:19.582489 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 13735
I20260925 19:07:19.582834 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.196.212:46595/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:38965
--local_ip_for_outbound_sockets=127.0.196.195
--tserver_master_addrs=127.0.196.254:39661
--webserver_port=46013
--webserver_interface=127.0.196.195
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.196.212:46595/kms with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 19:07:19 dist-test-slave-rn98 krb5kdc[12537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:07:19.592592 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:19.581292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:37483 (local address 127.0.196.254:39661)
0925 19:07:19.581339 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:19.581341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:19.581662 (+ 321us) server_negotiation.cc:407] Connection header received
0925 19:07:19.582688 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:19.582690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:19.582713 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:19.582792 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:19.583404 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:19.583974 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:19.585579 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:19.585933 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:19.585983 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:19.588679 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:19.588681 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:19.588682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:19.588691 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:19.590660 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:19.591265 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:19.591266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:19.591267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:19.591332 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:19.591667 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:19.591668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:19.591668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:19.591846 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:19.591896 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:19.592341 (+ 445us) server_negotiation.cc:299] Negotiation successful
0925 19:07:19.592449 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:07:19.593537 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:19.581315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:39661 (local address 127.0.196.194:37483)
0925 19:07:19.581498 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:19.581550 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:19.582181 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:19.582888 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:19.582893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:19.583085 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:07:19.583293 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:19.583297 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:19.584071 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:19.584072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:19.585467 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:19.585470 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:19.586096 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:19.586097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:19.586255 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:19.586955 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:07:19.586969 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:07:19.588542 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:07:19.590783 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:19.590792 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:19.590796 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:19.591161 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:19.591415 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:19.591416 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:19.591417 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:19.591581 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:19.591949 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:07:19.591955 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:07:19.592165 (+ 210us) client_negotiation.cc:769] Sending connection context
0925 19:07:19.593026 (+ 861us) client_negotiation.cc:240] Negotiation successful
0925 19:07:19.593231 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260925 19:07:19.594460 13865 heartbeater.cc:347] Connected to a master server at 127.0.196.254:39661
I20260925 19:07:19.594589 13865 heartbeater.cc:464] Registering TS with master...
I20260925 19:07:19.594837 13865 heartbeater.cc:511] Master 127.0.196.254:39661 requested a full tablet report, sending...
W20260925 19:07:19.595678 13865 heartbeater.cc:644] Failed to heartbeat to 127.0.196.254:39661 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 19:07:19.595764 13865 heartbeater.cc:464] Registering TS with master...
I20260925 19:07:19.595857 13865 heartbeater.cc:511] Master 127.0.196.254:39661 requested a full tablet report, sending...
I20260925 19:07:19.596266 13865 heartbeater.cc:464] Registering TS with master...
I20260925 19:07:19.596346 13865 heartbeater.cc:511] Master 127.0.196.254:39661 requested a full tablet report, sending...
W20260925 19:07:19.596678 13865 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 19:07:19.691953 13868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:07:19.692260 13868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:07:19.692302 13868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:07:19.694089 13868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:07:19.694157 13868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:07:19.694201 13868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:07:19.696501 13868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.196.212:46595/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:38965
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=46013
--tserver_master_addrs=127.0.196.254:39661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:07:19.696784 13868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:07:19.697106 13868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:07:19.703785 13874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:19.703805 13873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:19.703812 13876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:19.704659 13868 server_base.cc:1034] running on GCE node
I20260925 19:07:19.704885 13868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:07:19.705181 13868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:07:19.706341 13868 hybrid_clock.cc:648] HybridClock initialized: now 1790363239706325 us; error 34 us; skew 500 ppm
Sep 25 19:07:19 dist-test-slave-rn98 krb5kdc[12537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:19.711355 13868 init.cc:377] Logged in from keytab as kudu/127.0.196.195@KRBTEST.COM (short username kudu)
I20260925 19:07:19.711817 13868 webserver.cc:460] Webserver started at http://127.0.196.195:46013/ using document root <none> and password file <none>
I20260925 19:07:19.712064 13868 fs_manager.cc:359] Metadata directory not provided
I20260925 19:07:19.712149 13868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 19:07:19 dist-test-slave-rn98 krb5kdc[12537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for HTTP/127.0.196.212@KRBTEST.COM
I20260925 19:07:19.737924 13868 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.001s sys 0.000s
I20260925 19:07:19.740617 13884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:19.740842 13868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260925 19:07:19.740929 13868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "9fd3dfc7689746a39dfdd7f1de906b88"
format_stamp: "Formatted at 2026-09-25 19:07:18 on dist-test-slave-rn98"
server_key: "6c1308722d5dc9ca940a6946f5eedecd"
server_key_iv: "1d8d0ba5b83df941c7ec742bc2b01135"
server_key_version: "kuduclusterkey@0"
I20260925 19:07:19.741420 13868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:07:19.759406 13868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:07:19.762465 13868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:07:19.762668 13868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:07:19.762913 13868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:07:19.763262 13868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:07:19.763319 13868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:19.763362 13868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:07:19.763388 13868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:19.769477 13868 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:38965
I20260925 19:07:19.769563 13998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:38965 every 8 connection(s)
I20260925 19:07:19.769999 13868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
I20260925 19:07:19.777279 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 13868
Sep 25 19:07:19 dist-test-slave-rn98 krb5kdc[12537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:07:19.783001 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:19.770798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:36997 (local address 127.0.196.254:39661)
0925 19:07:19.770879 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:19.770881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:19.771154 (+ 273us) server_negotiation.cc:407] Connection header received
0925 19:07:19.771890 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:19.771891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:19.771902 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:19.771961 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:19.772737 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:19.773211 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:19.774767 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:19.774977 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:19.775059 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:19.778719 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:19.778721 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:19.778721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:19.778729 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:19.781184 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:19.781744 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:19.781746 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:19.781746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:19.781825 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:19.782162 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:19.782163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:19.782164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:19.782318 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:19.782391 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:19.782814 (+ 423us) server_negotiation.cc:299] Negotiation successful
0925 19:07:19.782901 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 19:07:19.783953 14001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:19.770773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:39661 (local address 127.0.196.195:36997)
0925 19:07:19.770995 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:19.771049 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:19.771740 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:19.772064 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:19.772071 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:19.772348 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:07:19.772642 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:19.772649 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:19.773316 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:19.773317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:19.774671 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:19.774674 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:19.775092 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:19.775093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:19.775197 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:19.776502 (+ 1305us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:07:19.776513 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:07:19.778552 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:07:19.781297 (+ 2745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:19.781306 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:19.781309 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:19.781645 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:19.781919 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:19.781920 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:19.781920 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:19.782083 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:19.782402 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:07:19.782409 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:07:19.782591 (+ 182us) client_negotiation.cc:769] Sending connection context
0925 19:07:19.783477 (+ 886us) client_negotiation.cc:240] Negotiation successful
0925 19:07:19.783672 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":118,"threads_started":1}
I20260925 19:07:19.784622 13999 heartbeater.cc:347] Connected to a master server at 127.0.196.254:39661
I20260925 19:07:19.784724 13999 heartbeater.cc:464] Registering TS with master...
I20260925 19:07:19.784926 13999 heartbeater.cc:511] Master 127.0.196.254:39661 requested a full tablet report, sending...
W20260925 19:07:19.785650 13999 heartbeater.cc:644] Failed to heartbeat to 127.0.196.254:39661 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 19:07:19.785728 13999 heartbeater.cc:464] Registering TS with master...
I20260925 19:07:19.785808 13999 heartbeater.cc:511] Master 127.0.196.254:39661 requested a full tablet report, sending...
I20260925 19:07:19.786177 13999 heartbeater.cc:464] Registering TS with master...
I20260925 19:07:19.786253 13999 heartbeater.cc:511] Master 127.0.196.254:39661 requested a full tablet report, sending...
W20260925 19:07:19.786638 13999 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 19:07:19.789120 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:19.778643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49522 (local address 127.0.196.254:39661)
0925 19:07:19.778770 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:19.778772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:19.778907 (+ 135us) server_negotiation.cc:407] Connection header received
0925 19:07:19.779230 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:19.779231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:19.779241 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:19.779323 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:19.780979 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:19.781790 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:19.784164 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:19.784491 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:19.784553 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:19.786116 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:19.786117 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:19.786117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:19.786124 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:19.787921 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:19.788335 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:19.788336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:19.788336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:19.788393 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:19.788588 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:19.788589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:19.788590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:19.788700 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:19.788889 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:19.788924 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 19:07:19.789005 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
Sep 25 19:07:20 dist-test-slave-rn98 krb5kdc[12537](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790363240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:20.408345 13731 heartbeater.cc:464] Registering TS with master...
I20260925 19:07:20.408516 13731 heartbeater.cc:511] Master 127.0.196.254:39661 requested a full tablet report, sending...
I20260925 19:07:20.409000 13533 ts_manager.cc:194] Registered new tserver with Master: c8c1e1a7bc7849d396b55a3bdab11c09 (127.0.196.193:43789)
I20260925 19:07:20.410063 13533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.193@KRBTEST.COM'} at 127.0.196.193:57019
I20260925 19:07:20.596881 13865 heartbeater.cc:464] Registering TS with master...
I20260925 19:07:20.597062 13865 heartbeater.cc:511] Master 127.0.196.254:39661 requested a full tablet report, sending...
I20260925 19:07:20.597558 13533 ts_manager.cc:194] Registered new tserver with Master: 338113a4b2014022ae4ff0ee8882217a (127.0.196.194:41761)
I20260925 19:07:20.598130 13533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.194@KRBTEST.COM'} at 127.0.196.194:37483
Sep 25 19:07:20 dist-test-slave-rn98 krb5kdc[12537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for HTTP/127.0.196.212@KRBTEST.COM
I20260925 19:07:20.786804 13999 heartbeater.cc:464] Registering TS with master...
I20260925 19:07:20.786990 13999 heartbeater.cc:511] Master 127.0.196.254:39661 requested a full tablet report, sending...
I20260925 19:07:20.787352 13533 ts_manager.cc:194] Registered new tserver with Master: 9fd3dfc7689746a39dfdd7f1de906b88 (127.0.196.195:38965)
I20260925 19:07:20.787853 13533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.195@KRBTEST.COM'} at 127.0.196.195:36997
I20260925 19:07:20.793183 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:07:20.802914 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:20.794191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49532 (local address 127.0.196.254:39661)
0925 19:07:20.794579 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:20.794582 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:20.794720 (+ 138us) server_negotiation.cc:407] Connection header received
0925 19:07:20.794815 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:20.794816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:20.794828 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:20.794870 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:20.795585 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:20.796421 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:20.797423 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:20.797731 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:20.797813 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:20.799383 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:20.799385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:20.799386 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:20.799394 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:20.801226 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:20.801674 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:20.801675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:20.801675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:20.801742 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:20.801931 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:20.801932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:20.801932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:20.802066 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:20.802205 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:20.802700 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 19:07:20.802797 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":67,"threads_started":1}
I20260925 19:07:20.805450 13533 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:07:20.807348 13533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:07:20.817960 14029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:20.810315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:43789 (local address 127.0.0.1:59298)
0925 19:07:20.810684 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:20.810730 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:20.810809 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:20.813694 (+ 2885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:20.813695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:20.813697 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:07:20.813738 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:20.813740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:20.815050 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:20.815051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:20.816710 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:20.816713 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:20.817662 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:20.817663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:20.817747 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:20.817760 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 19:07:20.817807 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 19:07:20.817868 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":236,"thread_start_us":86,"threads_started":1}
I20260925 19:07:20.818014 14030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:20.810585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38366 (local address 127.0.196.194:41761)
0925 19:07:20.810764 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:20.810770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:20.813208 (+ 2438us) server_negotiation.cc:407] Connection header received
0925 19:07:20.813298 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:20.813303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:20.813324 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:20.813421 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:07:20.813743 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:20.814947 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:20.816814 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:20.817559 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:20.817612 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:20.817683 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:20.817891 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 19:07:20.817928 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":75,"threads_started":1}
I20260925 19:07:20.818332 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:20.810351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:41761 (local address 127.0.0.1:38366)
0925 19:07:20.813114 (+ 2763us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:20.813143 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:20.813205 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:20.813534 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:20.813538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:20.813540 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:07:20.813626 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:20.813629 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:20.815048 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:20.815049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:20.816715 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:20.816716 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:20.817752 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:20.817752 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:20.817807 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:20.817810 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:07:20.818211 (+ 401us) client_negotiation.cc:240] Negotiation successful
0925 19:07:20.818254 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2744,"mutex_wait_us":277,"thread_start_us":89,"threads_started":1}
I20260925 19:07:20.818490 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:20.810550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:38965 (local address 127.0.0.1:36660)
0925 19:07:20.813312 (+ 2762us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:20.813321 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:20.813401 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:20.813689 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:20.813690 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:20.813693 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:07:20.813763 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:20.813765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:20.815124 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:20.815125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:20.816732 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:20.816735 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:20.817766 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:20.817767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:20.817838 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:20.817842 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:07:20.817891 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 19:07:20.817932 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2745,"mutex_wait_us":110,"thread_start_us":81,"threads_started":1}
I20260925 19:07:20.818558 14028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:20.810403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59298 (local address 127.0.196.193:43789)
0925 19:07:20.813531 (+ 3128us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:20.813536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:20.813547 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:07:20.813577 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:20.813578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:20.813592 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:20.813814 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:07:20.813885 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:20.814951 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:20.816808 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:20.817561 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:20.818304 (+ 743us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:20.818386 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:20.818415 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 19:07:20.818468 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3087,"thread_start_us":65,"threads_started":1}
I20260925 19:07:20.819247 13799 tablet_service.cc:1467] Processing CreateTablet for tablet 403b0833908b48a9a35ac191182b29e8 (DEFAULT_TABLE table=test-table [id=c6b2b52e4b284b41ab61101271798c95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:07:20.819396 13665 tablet_service.cc:1467] Processing CreateTablet for tablet 403b0833908b48a9a35ac191182b29e8 (DEFAULT_TABLE table=test-table [id=c6b2b52e4b284b41ab61101271798c95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:07:20.819641 13799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 403b0833908b48a9a35ac191182b29e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:07:20.819690 13665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 403b0833908b48a9a35ac191182b29e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:07:20.822192 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:20.810778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36660 (local address 127.0.196.195:38965)
0925 19:07:20.810930 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:20.810934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:20.813379 (+ 2445us) server_negotiation.cc:407] Connection header received
0925 19:07:20.813474 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:20.813477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:20.813495 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:20.813577 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:07:20.813852 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:20.815030 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:20.816828 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:20.817565 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:20.817635 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:20.817730 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:20.818003 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 19:07:20.818048 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260925 19:07:20.823194 13923 tablet_service.cc:1467] Processing CreateTablet for tablet 403b0833908b48a9a35ac191182b29e8 (DEFAULT_TABLE table=test-table [id=c6b2b52e4b284b41ab61101271798c95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:07:20.823539 13923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 403b0833908b48a9a35ac191182b29e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:07:20.826202 14033 tablet_bootstrap.cc:492] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88: Bootstrap starting.
I20260925 19:07:20.826763 14032 tablet_bootstrap.cc:492] T 403b0833908b48a9a35ac191182b29e8 P c8c1e1a7bc7849d396b55a3bdab11c09: Bootstrap starting.
I20260925 19:07:20.827105 14034 tablet_bootstrap.cc:492] T 403b0833908b48a9a35ac191182b29e8 P 338113a4b2014022ae4ff0ee8882217a: Bootstrap starting.
I20260925 19:07:20.827282 14033 tablet_bootstrap.cc:654] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88: Neither blocks nor log segments found. Creating new log.
I20260925 19:07:20.827476 14032 tablet_bootstrap.cc:654] T 403b0833908b48a9a35ac191182b29e8 P c8c1e1a7bc7849d396b55a3bdab11c09: Neither blocks nor log segments found. Creating new log.
I20260925 19:07:20.827656 14033 log.cc:824] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88: Log is configured to *not* fsync() on all Append() calls
I20260925 19:07:20.827841 14032 log.cc:824] T 403b0833908b48a9a35ac191182b29e8 P c8c1e1a7bc7849d396b55a3bdab11c09: Log is configured to *not* fsync() on all Append() calls
I20260925 19:07:20.827989 14034 tablet_bootstrap.cc:654] T 403b0833908b48a9a35ac191182b29e8 P 338113a4b2014022ae4ff0ee8882217a: Neither blocks nor log segments found. Creating new log.
I20260925 19:07:20.828298 14034 log.cc:824] T 403b0833908b48a9a35ac191182b29e8 P 338113a4b2014022ae4ff0ee8882217a: Log is configured to *not* fsync() on all Append() calls
I20260925 19:07:20.828768 14033 tablet_bootstrap.cc:492] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88: No bootstrap required, opened a new log
I20260925 19:07:20.828881 14033 ts_tablet_manager.cc:1397] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 19:07:20.828907 14032 tablet_bootstrap.cc:492] T 403b0833908b48a9a35ac191182b29e8 P c8c1e1a7bc7849d396b55a3bdab11c09: No bootstrap required, opened a new log
I20260925 19:07:20.828974 14032 ts_tablet_manager.cc:1397] T 403b0833908b48a9a35ac191182b29e8 P c8c1e1a7bc7849d396b55a3bdab11c09: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:07:20.828996 14034 tablet_bootstrap.cc:492] T 403b0833908b48a9a35ac191182b29e8 P 338113a4b2014022ae4ff0ee8882217a: No bootstrap required, opened a new log
I20260925 19:07:20.829087 14034 ts_tablet_manager.cc:1397] T 403b0833908b48a9a35ac191182b29e8 P 338113a4b2014022ae4ff0ee8882217a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:07:20.831975 14032 raft_consensus.cc:348] T 403b0833908b48a9a35ac191182b29e8 P c8c1e1a7bc7849d396b55a3bdab11c09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c1e1a7bc7849d396b55a3bdab11c09" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 43789 } } peers { permanent_uuid: "9fd3dfc7689746a39dfdd7f1de906b88" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 38965 } } peers { permanent_uuid: "338113a4b2014022ae4ff0ee8882217a" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 41761 } }
I20260925 19:07:20.832201 14032 raft_consensus.cc:374] T 403b0833908b48a9a35ac191182b29e8 P c8c1e1a7bc7849d396b55a3bdab11c09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:07:20.832252 14032 raft_consensus.cc:729] T 403b0833908b48a9a35ac191182b29e8 P c8c1e1a7bc7849d396b55a3bdab11c09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8c1e1a7bc7849d396b55a3bdab11c09, State: Initialized, Role: FOLLOWER
I20260925 19:07:20.832252 14033 raft_consensus.cc:348] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c1e1a7bc7849d396b55a3bdab11c09" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 43789 } } peers { permanent_uuid: "9fd3dfc7689746a39dfdd7f1de906b88" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 38965 } } peers { permanent_uuid: "338113a4b2014022ae4ff0ee8882217a" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 41761 } }
I20260925 19:07:20.832427 14033 raft_consensus.cc:374] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:07:20.832391 14032 consensus_queue.cc:260] T 403b0833908b48a9a35ac191182b29e8 P c8c1e1a7bc7849d396b55a3bdab11c09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c1e1a7bc7849d396b55a3bdab11c09" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 43789 } } peers { permanent_uuid: "9fd3dfc7689746a39dfdd7f1de906b88" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 38965 } } peers { permanent_uuid: "338113a4b2014022ae4ff0ee8882217a" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 41761 } }
I20260925 19:07:20.832496 14033 raft_consensus.cc:729] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fd3dfc7689746a39dfdd7f1de906b88, State: Initialized, Role: FOLLOWER
I20260925 19:07:20.832132 14034 raft_consensus.cc:348] T 403b0833908b48a9a35ac191182b29e8 P 338113a4b2014022ae4ff0ee8882217a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c1e1a7bc7849d396b55a3bdab11c09" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 43789 } } peers { permanent_uuid: "9fd3dfc7689746a39dfdd7f1de906b88" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 38965 } } peers { permanent_uuid: "338113a4b2014022ae4ff0ee8882217a" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 41761 } }
I20260925 19:07:20.832656 14034 raft_consensus.cc:374] T 403b0833908b48a9a35ac191182b29e8 P 338113a4b2014022ae4ff0ee8882217a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:07:20.832695 14032 ts_tablet_manager.cc:1428] T 403b0833908b48a9a35ac191182b29e8 P c8c1e1a7bc7849d396b55a3bdab11c09: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260925 19:07:20.832665 14033 consensus_queue.cc:260] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c1e1a7bc7849d396b55a3bdab11c09" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 43789 } } peers { permanent_uuid: "9fd3dfc7689746a39dfdd7f1de906b88" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 38965 } } peers { permanent_uuid: "338113a4b2014022ae4ff0ee8882217a" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 41761 } }
I20260925 19:07:20.832852 13731 heartbeater.cc:503] Master 127.0.196.254:39661 was elected leader, sending a full tablet report...
I20260925 19:07:20.832945 14033 ts_tablet_manager.cc:1428] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 19:07:20.833040 13999 heartbeater.cc:503] Master 127.0.196.254:39661 was elected leader, sending a full tablet report...
I20260925 19:07:20.832736 14034 raft_consensus.cc:729] T 403b0833908b48a9a35ac191182b29e8 P 338113a4b2014022ae4ff0ee8882217a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 338113a4b2014022ae4ff0ee8882217a, State: Initialized, Role: FOLLOWER
I20260925 19:07:20.833547 14034 consensus_queue.cc:260] T 403b0833908b48a9a35ac191182b29e8 P 338113a4b2014022ae4ff0ee8882217a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c1e1a7bc7849d396b55a3bdab11c09" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 43789 } } peers { permanent_uuid: "9fd3dfc7689746a39dfdd7f1de906b88" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 38965 } } peers { permanent_uuid: "338113a4b2014022ae4ff0ee8882217a" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 41761 } }
I20260925 19:07:20.833864 14034 ts_tablet_manager.cc:1428] T 403b0833908b48a9a35ac191182b29e8 P 338113a4b2014022ae4ff0ee8882217a: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260925 19:07:20.834010 13865 heartbeater.cc:503] Master 127.0.196.254:39661 was elected leader, sending a full tablet report...
I20260925 19:07:20.857247 14039 raft_consensus.cc:482] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:07:20.857419 14039 raft_consensus.cc:504] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c1e1a7bc7849d396b55a3bdab11c09" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 43789 } } peers { permanent_uuid: "9fd3dfc7689746a39dfdd7f1de906b88" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 38965 } } peers { permanent_uuid: "338113a4b2014022ae4ff0ee8882217a" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 41761 } }
I20260925 19:07:20.857833 14039 leader_election.cc:288] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8c1e1a7bc7849d396b55a3bdab11c09 (127.0.196.193:43789), 338113a4b2014022ae4ff0ee8882217a (127.0.196.194:41761)
I20260925 19:07:20.862150 14030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:20.858225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:46445 (local address 127.0.196.194:41761)
0925 19:07:20.858274 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:20.858277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:20.858407 (+ 130us) server_negotiation.cc:407] Connection header received
0925 19:07:20.858537 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:20.858538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:20.858550 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:20.858636 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:07:20.858869 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:20.859633 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:20.861178 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:20.861746 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:20.861802 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:20.861812 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:20.862026 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 19:07:20.862071 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:07:20.862267 14028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:20.858415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:50871 (local address 127.0.196.193:43789)
0925 19:07:20.858458 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:20.858460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:20.858787 (+ 327us) server_negotiation.cc:407] Connection header received
0925 19:07:20.858881 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:20.858882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:20.858891 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:20.858918 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:07:20.859160 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:20.859931 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:20.861298 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:20.861877 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:20.861924 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:20.861931 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:20.862148 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 19:07:20.862188 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:07:20.862499 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:20.858203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:43789 (local address 127.0.196.195:50871)
0925 19:07:20.858729 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:20.858740 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:20.858812 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:20.858986 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:20.858987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:20.858989 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:07:20.859074 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:20.859076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:20.860032 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:20.860033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:20.861185 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:20.861188 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:20.862003 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:20.862004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:20.862077 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:20.862080 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:07:20.862376 (+ 296us) client_negotiation.cc:240] Negotiation successful
0925 19:07:20.862419 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":99,"threads_started":1}
I20260925 19:07:20.862793 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:20.858036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:41761 (local address 127.0.196.195:46445)
0925 19:07:20.858338 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:20.858349 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:20.858435 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:20.858668 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:20.858669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:20.858672 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:07:20.858773 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:20.858775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:20.859741 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:20.859741 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:20.861062 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:20.861065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:20.861875 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:20.861876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:20.861946 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:20.861950 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:07:20.862657 (+ 707us) client_negotiation.cc:240] Negotiation successful
0925 19:07:20.862707 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":110,"threads_started":1}
I20260925 19:07:20.862922 13685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "403b0833908b48a9a35ac191182b29e8" candidate_uuid: "9fd3dfc7689746a39dfdd7f1de906b88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8c1e1a7bc7849d396b55a3bdab11c09" is_pre_election: true
I20260925 19:07:20.863106 13685 raft_consensus.cc:2454] T 403b0833908b48a9a35ac191182b29e8 P c8c1e1a7bc7849d396b55a3bdab11c09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fd3dfc7689746a39dfdd7f1de906b88 in term 0.
I20260925 19:07:20.863081 13800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "403b0833908b48a9a35ac191182b29e8" candidate_uuid: "9fd3dfc7689746a39dfdd7f1de906b88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "338113a4b2014022ae4ff0ee8882217a" is_pre_election: true
I20260925 19:07:20.863224 13800 raft_consensus.cc:2454] T 403b0833908b48a9a35ac191182b29e8 P 338113a4b2014022ae4ff0ee8882217a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fd3dfc7689746a39dfdd7f1de906b88 in term 0.
I20260925 19:07:20.863351 13885 leader_election.cc:302] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fd3dfc7689746a39dfdd7f1de906b88, c8c1e1a7bc7849d396b55a3bdab11c09; no voters:
I20260925 19:07:20.863480 14039 raft_consensus.cc:2781] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:07:20.863550 14039 raft_consensus.cc:482] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:07:20.863582 14039 raft_consensus.cc:3037] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:07:20.864454 14039 raft_consensus.cc:504] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c1e1a7bc7849d396b55a3bdab11c09" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 43789 } } peers { permanent_uuid: "9fd3dfc7689746a39dfdd7f1de906b88" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 38965 } } peers { permanent_uuid: "338113a4b2014022ae4ff0ee8882217a" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 41761 } }
I20260925 19:07:20.864583 14039 leader_election.cc:288] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88 [CANDIDATE]: Term 1 election: Requested vote from peers c8c1e1a7bc7849d396b55a3bdab11c09 (127.0.196.193:43789), 338113a4b2014022ae4ff0ee8882217a (127.0.196.194:41761)
I20260925 19:07:20.864784 13800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "403b0833908b48a9a35ac191182b29e8" candidate_uuid: "9fd3dfc7689746a39dfdd7f1de906b88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "338113a4b2014022ae4ff0ee8882217a"
I20260925 19:07:20.864857 13685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "403b0833908b48a9a35ac191182b29e8" candidate_uuid: "9fd3dfc7689746a39dfdd7f1de906b88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8c1e1a7bc7849d396b55a3bdab11c09"
I20260925 19:07:20.864879 13800 raft_consensus.cc:3037] T 403b0833908b48a9a35ac191182b29e8 P 338113a4b2014022ae4ff0ee8882217a [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:07:20.864934 13685 raft_consensus.cc:3037] T 403b0833908b48a9a35ac191182b29e8 P c8c1e1a7bc7849d396b55a3bdab11c09 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:07:20.865685 13800 raft_consensus.cc:2454] T 403b0833908b48a9a35ac191182b29e8 P 338113a4b2014022ae4ff0ee8882217a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fd3dfc7689746a39dfdd7f1de906b88 in term 1.
I20260925 19:07:20.865860 13685 raft_consensus.cc:2454] T 403b0833908b48a9a35ac191182b29e8 P c8c1e1a7bc7849d396b55a3bdab11c09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fd3dfc7689746a39dfdd7f1de906b88 in term 1.
I20260925 19:07:20.866010 13885 leader_election.cc:302] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fd3dfc7689746a39dfdd7f1de906b88, c8c1e1a7bc7849d396b55a3bdab11c09; no voters:
I20260925 19:07:20.866117 14039 raft_consensus.cc:2781] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:07:20.866284 14039 raft_consensus.cc:686] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88 [term 1 LEADER]: Becoming Leader. State: Replica: 9fd3dfc7689746a39dfdd7f1de906b88, State: Running, Role: LEADER
I20260925 19:07:20.866369 14039 consensus_queue.cc:237] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c1e1a7bc7849d396b55a3bdab11c09" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 43789 } } peers { permanent_uuid: "9fd3dfc7689746a39dfdd7f1de906b88" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 38965 } } peers { permanent_uuid: "338113a4b2014022ae4ff0ee8882217a" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 41761 } }
I20260925 19:07:20.867177 13532 catalog_manager.cc:5495] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fd3dfc7689746a39dfdd7f1de906b88 (127.0.196.195). New cstate: current_term: 1 leader_uuid: "9fd3dfc7689746a39dfdd7f1de906b88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8c1e1a7bc7849d396b55a3bdab11c09" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 43789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fd3dfc7689746a39dfdd7f1de906b88" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 38965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "338113a4b2014022ae4ff0ee8882217a" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 41761 } health_report { overall_health: UNKNOWN } } }
I20260925 19:07:20.873904 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:20.870904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36664 (local address 127.0.196.195:38965)
0925 19:07:20.870946 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:20.870948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:20.870956 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:07:20.870992 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:20.870993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:20.871013 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:20.871087 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:07:20.871255 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:20.871926 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:20.873046 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:20.873325 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:20.873388 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:20.873555 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:07:20.873680 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:07:20.873757 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:20.873813 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 19:07:20.873834 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:07:20.879832 13800 raft_consensus.cc:1261] T 403b0833908b48a9a35ac191182b29e8 P 338113a4b2014022ae4ff0ee8882217a [term 1 FOLLOWER]: Refusing update from remote peer 9fd3dfc7689746a39dfdd7f1de906b88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:07:20.879838 13685 raft_consensus.cc:1261] T 403b0833908b48a9a35ac191182b29e8 P c8c1e1a7bc7849d396b55a3bdab11c09 [term 1 FOLLOWER]: Refusing update from remote peer 9fd3dfc7689746a39dfdd7f1de906b88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:07:20.880249 14043 consensus_queue.cc:1035] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "338113a4b2014022ae4ff0ee8882217a" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 41761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:07:20.880366 14039 consensus_queue.cc:1035] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8c1e1a7bc7849d396b55a3bdab11c09" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 43789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:07:20.884137 14048 mvcc.cc:205] Tried to move back new op lower bound from 7333327834640674816 to 7333327834589126656. Current Snapshot: MvccSnapshot[applied={T|T < 7333327834640674816}]
I20260925 19:07:20.884770 14046 mvcc.cc:205] Tried to move back new op lower bound from 7333327834640674816 to 7333327834589126656. Current Snapshot: MvccSnapshot[applied={T|T < 7333327834640674816}]
I20260925 19:07:20.884876 14047 mvcc.cc:205] Tried to move back new op lower bound from 7333327834640674816 to 7333327834589126656. Current Snapshot: MvccSnapshot[applied={T|T < 7333327834640674816}]
I20260925 19:07:20.887856 13533 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:07:20.887959 13533 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:07:20.888760 13533 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5399492cbf3d4c75ac32f1d05c70f2fe: Sending DeleteTablet for 3 replicas of tablet 403b0833908b48a9a35ac191182b29e8
I20260925 19:07:20.889168 13923 tablet_service.cc:1514] Processing DeleteTablet for tablet 403b0833908b48a9a35ac191182b29e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:07:20 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:36660
I20260925 19:07:20.889168 13665 tablet_service.cc:1514] Processing DeleteTablet for tablet 403b0833908b48a9a35ac191182b29e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:07:20 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:59298
I20260925 19:07:20.889329 13799 tablet_service.cc:1514] Processing DeleteTablet for tablet 403b0833908b48a9a35ac191182b29e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:07:20 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:38366
I20260925 19:07:20.889778 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 13571
I20260925 19:07:20.895236 14057 tablet_replica.cc:330] T 403b0833908b48a9a35ac191182b29e8 P 338113a4b2014022ae4ff0ee8882217a: stopping tablet replica
I20260925 19:07:20.896164 14057 raft_consensus.cc:2229] T 403b0833908b48a9a35ac191182b29e8 P 338113a4b2014022ae4ff0ee8882217a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:07:20.896464 14057 raft_consensus.cc:2258] T 403b0833908b48a9a35ac191182b29e8 P 338113a4b2014022ae4ff0ee8882217a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 19:07:20.898516 13517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.196.193:43789 (error 108)
I20260925 19:07:20.898666 14055 tablet_replica.cc:330] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88: stopping tablet replica
I20260925 19:07:20.898787 14055 raft_consensus.cc:2229] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:07:20.898936 14055 raft_consensus.cc:2258] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:07:20.899145 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 13735
I20260925 19:07:20.900936 14055 ts_tablet_manager.cc:1905] T 403b0833908b48a9a35ac191182b29e8 P 9fd3dfc7689746a39dfdd7f1de906b88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:07:20.903112 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:20.902936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:43789 (local address 127.0.0.1:59306)
0925 19:07:20.903023 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:20.903068 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.193:43789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 19:07:20.903386 13517 catalog_manager.cc:4584] TS c8c1e1a7bc7849d396b55a3bdab11c09 (127.0.196.193:43789): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 403b0833908b48a9a35ac191182b29e8: Network error: Client connection negotiation failed: client connection to 127.0.196.193:43789: connect: Connection refused (error 111)
W20260925 19:07:20.907819 14026 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:07:20.907496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:41761 (local address 127.0.0.1:38370)
0925 19:07:20.907552 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:20.907564 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:20.907610 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:20.907760 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.194:41761: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 19:07:20.908458 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 13868
I20260925 19:07:20.914091 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:20.913969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:38965 (local address 127.0.0.1:36668)
0925 19:07:20.914019 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:20.914047 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.195:38965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 19:07:20.914163 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 13500
I20260925 19:07:20.921619 787 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 19:07:21.266374 787 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 19:07:21.266472 787 mini_ranger.cc:66] Stopping Ranger...
I20260925 19:07:21.611603 787 mini_ranger.cc:68] Stopped Ranger
2026-09-25 19:07:21.611 UTC [12567] LOG: received smart shutdown request
2026-09-25 19:07:21.615 UTC [12567] LOG: background worker "logical replication launcher" (PID 12576) exited with exit code 1
2026-09-25 19:07:21.616 UTC [12571] LOG: shutting down
2026-09-25 19:07:21.801 UTC [12567] LOG: database system is shut down
2026-09-25T19:07:21Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49949 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:07:21 dist-test-slave-rn98 krb5kdc[14067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:07:21 dist-test-slave-rn98 krb5kdc[14067](info): set up 2 sockets
Sep 25 19:07:21 dist-test-slave-rn98 krb5kdc[14067](info): commencing operation
krb5kdc: starting...
W20260925 19:07:22.946591 787 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:07:23 dist-test-slave-rn98 krb5kdc[14067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:07:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:07:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:07:23.069604 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:37375
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:33271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:37375,127.0.196.253:37255,127.0.196.252:38513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:07:23.152556 14080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:07:23.152793 14080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:07:23.152818 14080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:07:23.153935 14080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:07:23.153973 14080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:07:23.153988 14080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:07:23.154000 14080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:07:23.154012 14080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:07:23.155660 14080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:33271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:37375,127.0.196.253:37255,127.0.196.252:38513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:37375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:07:23.155957 14080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:07:23.156207 14080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:07:23.161345 14085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:23.161357 14086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:23.161365 14088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:23.161777 14080 server_base.cc:1034] running on GCE node
I20260925 19:07:23.162062 14080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:07:23.162379 14080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:07:23.163585 14080 hybrid_clock.cc:648] HybridClock initialized: now 1790363243163575 us; error 39 us; skew 500 ppm
Sep 25 19:07:23 dist-test-slave-rn98 krb5kdc[14067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:23.170102 14080 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:07:23.170661 14080 webserver.cc:460] Webserver started at http://127.0.196.254:40405/ using document root <none> and password file <none>
I20260925 19:07:23.170924 14080 fs_manager.cc:359] Metadata directory not provided
I20260925 19:07:23.170996 14080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:07:23.171157 14080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:07:23.172271 14080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "674f387cddb3491dbfdaa8a595cd0049"
format_stamp: "Formatted at 2026-09-25 19:07:23 on dist-test-slave-rn98"
server_key: "cd7e656182ba0a7fb961123637931ec1"
server_key_iv: "70df91ba0ee484d143fd9e3c37d48916"
server_key_version: "encryptionkey@0"
I20260925 19:07:23.172650 14080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "674f387cddb3491dbfdaa8a595cd0049"
format_stamp: "Formatted at 2026-09-25 19:07:23 on dist-test-slave-rn98"
server_key: "cd7e656182ba0a7fb961123637931ec1"
server_key_iv: "70df91ba0ee484d143fd9e3c37d48916"
server_key_version: "encryptionkey@0"
I20260925 19:07:23.174132 14080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:07:23.175127 14095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:23.175339 14080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:07:23.175459 14080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "674f387cddb3491dbfdaa8a595cd0049"
format_stamp: "Formatted at 2026-09-25 19:07:23 on dist-test-slave-rn98"
server_key: "cd7e656182ba0a7fb961123637931ec1"
server_key_iv: "70df91ba0ee484d143fd9e3c37d48916"
server_key_version: "encryptionkey@0"
I20260925 19:07:23.175537 14080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:07:23.194016 14080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:07:23.196535 14080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:07:23.196709 14080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:07:23.200963 14080 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:37375
I20260925 19:07:23.201006 14147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:37375 every 8 connection(s)
I20260925 19:07:23.201399 14080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:07:23.202414 14148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:07:23.204066 14148 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37375 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:07:23.204923 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 14080
I20260925 19:07:23.205024 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:07:23.205255 787 external_mini_cluster.cc:1428] Setting key e7544f4ba8902055934b381c1db934eb
Sep 25 19:07:23 dist-test-slave-rn98 krb5kdc[14067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.196.253@KRBTEST.COM; defaulting to no policy
I20260925 19:07:23.214568 14150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.204831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:37375 (local address 127.0.0.1:41690)
0925 19:07:23.205012 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:23.205056 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:23.205595 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:23.205932 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:23.205935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:23.206194 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:07:23.206367 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:23.206371 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.207514 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.207515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:23.208208 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:23.208210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.208720 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.208721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:23.208784 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:23.209361 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:07:23.209369 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:07:23.211134 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:07:23.213030 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:23.213036 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:23.213038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:23.213239 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:23.213488 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:23.213489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:23.213489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:23.213583 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:23.214001 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:07:23.214006 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:07:23.214065 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 19:07:23.214148 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 19:07:23.214338 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":101,"threads_started":1}
I20260925 19:07:23.214572 14151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.205048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41690 (local address 127.0.196.254:37375)
0925 19:07:23.205194 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:23.205200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:23.205243 (+ 43us) server_negotiation.cc:407] Connection header received
0925 19:07:23.205774 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:23.205777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:23.205800 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:23.205910 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:23.206473 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.207403 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.208292 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.208622 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.208749 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:23.211247 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:23.211254 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:23.211254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:23.211261 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:23.212944 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.213336 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.213339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.213339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.213419 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.213683 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.213684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.213684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.213928 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:23.214004 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:23.214237 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 19:07:23.214399 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.253@KRBTEST.COM" created.
I20260925 19:07:23.215596 14148 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.253" port: 37255 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:07:23.216043 14150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.215903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:37255 (local address 127.0.0.1:48944)
0925 19:07:23.215955 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:23.216008 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.253:37255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 19:07:23.216207 14098 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.196.253:37255: connect: Connection refused (error 111)
I20260925 19:07:23.216629 14150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.216518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:37255 (local address 127.0.0.1:48950)
0925 19:07:23.216561 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:23.216591 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.253:37255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 19:07:23.216789 14148 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.196.253:37255: Network error: Client connection negotiation failed: client connection to 127.0.196.253:37255: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.253.keytab.
Entry for principal kudu/127.0.196.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.253.keytab.
I20260925 19:07:23.229758 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.253:37255
--webserver_interface=127.0.196.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.253.keytab
--principal=kudu/127.0.196.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:33271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:37375,127.0.196.253:37255,127.0.196.252:38513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:07:23.266065 14148 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.253" port: 37255 } attempt: 1
I20260925 19:07:23.266685 14150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.266440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:37255 (local address 127.0.0.1:54858)
0925 19:07:23.266543 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:23.266608 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.253:37255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 19:07:23.267158 14150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.267020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:37255 (local address 127.0.0.1:54874)
0925 19:07:23.267078 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:23.267125 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.253:37255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 19:07:23.267303 14148 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.196.253:37255: Network error: Client connection negotiation failed: client connection to 127.0.196.253:37255: connect: Connection refused (error 111)
W20260925 19:07:23.314913 14155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:07:23.315148 14155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:07:23.315172 14155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:07:23.316424 14155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:07:23.316481 14155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:07:23.316497 14155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:07:23.316510 14155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:07:23.316522 14155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:07:23.318128 14155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:33271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:37375,127.0.196.253:37255,127.0.196.252:38513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.253.keytab
--principal=kudu/127.0.196.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.253:37255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.196.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:07:23.318405 14155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:07:23.318686 14155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:07:23.324574 14163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:23.324569 14161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:23.324604 14155 server_base.cc:1034] running on GCE node
W20260925 19:07:23.324569 14160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:23.325049 14155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:07:23.325342 14155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:07:23.326498 14155 hybrid_clock.cc:648] HybridClock initialized: now 1790363243326476 us; error 40 us; skew 500 ppm
Sep 25 19:07:23 dist-test-slave-rn98 krb5kdc[14067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:23.329317 14155 init.cc:377] Logged in from keytab as kudu/127.0.196.253@KRBTEST.COM (short username kudu)
I20260925 19:07:23.329882 14155 webserver.cc:460] Webserver started at http://127.0.196.253:34359/ using document root <none> and password file <none>
I20260925 19:07:23.330121 14155 fs_manager.cc:359] Metadata directory not provided
I20260925 19:07:23.330191 14155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:07:23.330351 14155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:07:23.331528 14155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-1/data/instance:
uuid: "92dad4d16cc1493a9c1d27b1e6941c5c"
format_stamp: "Formatted at 2026-09-25 19:07:23 on dist-test-slave-rn98"
server_key: "cefdf4ec0ecae4dbe2893b78e9cb8288"
server_key_iv: "40f3ad69ba30ab0449dbecdcef25ee4d"
server_key_version: "encryptionkey@0"
I20260925 19:07:23.331899 14155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-1/wal/instance:
uuid: "92dad4d16cc1493a9c1d27b1e6941c5c"
format_stamp: "Formatted at 2026-09-25 19:07:23 on dist-test-slave-rn98"
server_key: "cefdf4ec0ecae4dbe2893b78e9cb8288"
server_key_iv: "40f3ad69ba30ab0449dbecdcef25ee4d"
server_key_version: "encryptionkey@0"
I20260925 19:07:23.333343 14155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 19:07:23.334280 14170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:23.334604 14155 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:07:23.334733 14155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-1/wal
uuid: "92dad4d16cc1493a9c1d27b1e6941c5c"
format_stamp: "Formatted at 2026-09-25 19:07:23 on dist-test-slave-rn98"
server_key: "cefdf4ec0ecae4dbe2893b78e9cb8288"
server_key_iv: "40f3ad69ba30ab0449dbecdcef25ee4d"
server_key_version: "encryptionkey@0"
I20260925 19:07:23.334822 14155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:07:23.335475 14148 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.253" port: 37255 } attempt: 2
I20260925 19:07:23.335960 14150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.335820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:37255 (local address 127.0.0.1:54888)
0925 19:07:23.335884 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:23.335919 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.253:37255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 19:07:23.336338 14150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.336237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:37255 (local address 127.0.0.1:54896)
0925 19:07:23.336287 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:23.336313 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.253:37255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 19:07:23.336439 14148 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.196.253:37255: Network error: Client connection negotiation failed: client connection to 127.0.196.253:37255: connect: Connection refused (error 111)
I20260925 19:07:23.351882 14155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:07:23.354315 14155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:07:23.354496 14155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:07:23.358703 14155 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.253:37255
I20260925 19:07:23.358827 14222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.253:37255 every 8 connection(s)
I20260925 19:07:23.359154 14155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-1/data/info.pb
I20260925 19:07:23.360114 14223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:07:23.361752 14223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37375 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:07:23.365061 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 14155
I20260925 19:07:23.365182 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-1/wal/instance
I20260925 19:07:23.365386 787 external_mini_cluster.cc:1428] Setting key e4d7dec624e0cef1c8a31152c3e1a8a2
Sep 25 19:07:23 dist-test-slave-rn98 krb5kdc[14067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.253@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:07:23.371975 14151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.362518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59198 (local address 127.0.196.254:37375)
0925 19:07:23.362637 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:23.362639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:23.362817 (+ 178us) server_negotiation.cc:407] Connection header received
0925 19:07:23.363334 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:23.363335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:23.363350 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:23.363411 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:23.364095 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.364595 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.365774 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.365987 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.366064 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:23.368726 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:23.368728 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:23.368728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:23.368738 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:23.370289 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.370848 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.370850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.370850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.370945 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.371250 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.371251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.371251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.371380 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:23.371458 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:23.371754 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 19:07:23.371878 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 19:07:23.372161 14225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.362477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:37375 (local address 127.0.0.1:59198)
0925 19:07:23.362676 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:23.362727 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:23.363223 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:23.363489 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:23.363495 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:23.363756 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:07:23.363958 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:23.363963 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.364693 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.364694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:23.365686 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:23.365689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.366074 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.366075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:23.366210 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:23.366849 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:07:23.366862 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:07:23.368607 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:07:23.370420 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:23.370429 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:23.370431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:23.370768 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:23.371021 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:23.371023 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:23.371023 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:23.371152 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:23.371449 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:07:23.371457 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:07:23.371614 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 19:07:23.371726 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 19:07:23.371908 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":79,"threads_started":1}
I20260925 19:07:23.372749 14223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.253" port: 37255 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.196.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.252@KRBTEST.COM" created.
Sep 25 19:07:23 dist-test-slave-rn98 krb5kdc[14067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.253@KRBTEST.COM for kudu/127.0.196.253@KRBTEST.COM
I20260925 19:07:23.380942 14225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.373054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:37255 (local address 127.0.0.1:54898)
0925 19:07:23.373281 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:23.373292 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:23.373351 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:23.373516 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:23.373517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:23.373662 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:07:23.373710 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:23.373711 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.374656 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.374657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:23.375171 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:23.375173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.375692 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.375693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:23.375766 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:23.376310 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:07:23.376314 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:07:23.377520 (+ 1206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:07:23.379740 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:23.379741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:23.379741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:23.379909 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:23.380171 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:23.380173 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:23.380173 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:23.380252 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:23.380667 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:07:23.380668 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:07:23.380717 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 19:07:23.380786 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 19:07:23.380871 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211}
I20260925 19:07:23.381101 14227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.373137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54898 (local address 127.0.196.253:37255)
0925 19:07:23.373290 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:23.373294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:23.373343 (+ 49us) server_negotiation.cc:407] Connection header received
0925 19:07:23.373408 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:23.373409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:23.373421 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:23.373502 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:23.373788 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.374539 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.375277 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.375603 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.375693 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:23.377608 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:23.377618 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:23.377618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:23.377625 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:23.379643 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.379999 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.380003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.380003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.380070 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.380347 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.380348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.380348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.380576 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:23.380668 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:23.380848 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 19:07:23.380940 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
I20260925 19:07:23.381943 14223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.252" port: 38513 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:07:23.382375 14225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.382225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:38513 (local address 127.0.0.1:48556)
0925 19:07:23.382288 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:23.382340 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 19:07:23.382529 14174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.196.252:38513: connect: Connection refused (error 111)
I20260925 19:07:23.383090 14225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.382979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:38513 (local address 127.0.0.1:48560)
0925 19:07:23.383032 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:23.383065 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 19:07:23.383260 14223 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.196.252:38513: Network error: Client connection negotiation failed: client connection to 127.0.196.252:38513: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.252.keytab.
Entry for principal kudu/127.0.196.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.252.keytab.
I20260925 19:07:23.393939 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.252:38513
--webserver_interface=127.0.196.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.252.keytab
--principal=kudu/127.0.196.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:33271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:37375,127.0.196.253:37255,127.0.196.252:38513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:07:23.427591 14148 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.253" port: 37255 } attempt: 3
Sep 25 19:07:23 dist-test-slave-rn98 krb5kdc[14067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for kudu/127.0.196.253@KRBTEST.COM
I20260925 19:07:23.432574 14223 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.252" port: 38513 } attempt: 1
I20260925 19:07:23.433071 14225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.432921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:38513 (local address 127.0.0.1:48562)
0925 19:07:23.432991 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:23.433030 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 19:07:23.433434 14225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.433307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:38513 (local address 127.0.0.1:48568)
0925 19:07:23.433361 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:23.433397 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 19:07:23.433701 14223 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.196.252:38513: Network error: Client connection negotiation failed: client connection to 127.0.196.252:38513: connect: Connection refused (error 111)
I20260925 19:07:23.435925 14227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.428141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54910 (local address 127.0.196.253:37255)
0925 19:07:23.428220 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:23.428222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:23.428236 (+ 14us) server_negotiation.cc:407] Connection header received
0925 19:07:23.428301 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:23.428302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:23.428316 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:23.428381 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:23.428692 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.429471 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.430224 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.430573 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.430666 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:23.432940 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:23.432942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:23.432942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:23.432950 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:23.434743 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.435158 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.435159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.435160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.435212 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.435443 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.435444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.435444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.435550 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:23.435600 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:23.435802 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 19:07:23.435869 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 19:07:23.436046 14150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.428074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:37255 (local address 127.0.0.1:54910)
0925 19:07:23.428168 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:23.428181 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:23.428246 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:23.428394 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:23.428394 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:23.428543 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:07:23.428612 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:23.428613 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.429575 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.429576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:23.430131 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:23.430133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.430723 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.430724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:23.430795 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:23.431323 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:07:23.431327 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:07:23.432832 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:07:23.434829 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:23.434830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:23.434831 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:23.435061 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:23.435282 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:23.435283 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:23.435284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:23.435371 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:23.435620 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:07:23.435621 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:07:23.435689 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 19:07:23.435778 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 19:07:23.435909 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 19:07:23.436542 14148 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.252" port: 38513 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:07:23.436911 14150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.436822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:38513 (local address 127.0.0.1:48572)
0925 19:07:23.436859 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:23.436889 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 19:07:23.437206 14150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.437132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:38513 (local address 127.0.0.1:48578)
0925 19:07:23.437159 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:23.437181 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260925 19:07:23.437292 14148 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.196.252:38513: Network error: Client connection negotiation failed: client connection to 127.0.196.252:38513: connect: Connection refused (error 111)
I20260925 19:07:23.468439 14148 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.252" port: 38513 } attempt: 1
I20260925 19:07:23.468950 14150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.468804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:38513 (local address 127.0.0.1:48582)
0925 19:07:23.468875 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:23.468917 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 19:07:23.469307 14150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.469211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:38513 (local address 127.0.0.1:48594)
0925 19:07:23.469256 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:23.469283 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 19:07:23.469394 14148 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.196.252:38513: Network error: Client connection negotiation failed: client connection to 127.0.196.252:38513: connect: Connection refused (error 111)
W20260925 19:07:23.487450 14230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:07:23.487735 14230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:07:23.487762 14230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:07:23.489094 14230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:07:23.489143 14230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:07:23.489158 14230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:07:23.489171 14230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:07:23.489184 14230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:07:23.490901 14230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:33271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:37375,127.0.196.253:37255,127.0.196.252:38513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.252.keytab
--principal=kudu/127.0.196.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.252:38513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.196.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:07:23.491185 14230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:07:23.491442 14230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:07:23.496480 14235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:23.496466 14238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:23.496487 14236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:23.496834 14230 server_base.cc:1034] running on GCE node
I20260925 19:07:23.497120 14230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:07:23.497433 14230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:07:23.498673 14230 hybrid_clock.cc:648] HybridClock initialized: now 1790363243498611 us; error 46 us; skew 500 ppm
Sep 25 19:07:23 dist-test-slave-rn98 krb5kdc[14067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:23.501428 14230 init.cc:377] Logged in from keytab as kudu/127.0.196.252@KRBTEST.COM (short username kudu)
I20260925 19:07:23.501929 14230 webserver.cc:460] Webserver started at http://127.0.196.252:43959/ using document root <none> and password file <none>
I20260925 19:07:23.501850 14223 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.252" port: 38513 } attempt: 2
I20260925 19:07:23.502194 14230 fs_manager.cc:359] Metadata directory not provided
I20260925 19:07:23.502266 14230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:07:23.502334 14225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.502176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:38513 (local address 127.0.0.1:48602)
0925 19:07:23.502238 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:23.502297 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 19:07:23.502445 14230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:07:23.502818 14225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.502718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:38513 (local address 127.0.0.1:48610)
0925 19:07:23.502765 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:23.502800 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 19:07:23.502939 14223 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.196.252:38513: Network error: Client connection negotiation failed: client connection to 127.0.196.252:38513: connect: Connection refused (error 111)
I20260925 19:07:23.503759 14230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-2/data/instance:
uuid: "55147c62bbda4e6da02eaa96a928c04e"
format_stamp: "Formatted at 2026-09-25 19:07:23 on dist-test-slave-rn98"
server_key: "23f60bef5f3a451db0e67f0a21243873"
server_key_iv: "6659e743102220c664e4cdb3a9300a04"
server_key_version: "encryptionkey@0"
I20260925 19:07:23.504134 14230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-2/wal/instance:
uuid: "55147c62bbda4e6da02eaa96a928c04e"
format_stamp: "Formatted at 2026-09-25 19:07:23 on dist-test-slave-rn98"
server_key: "23f60bef5f3a451db0e67f0a21243873"
server_key_iv: "6659e743102220c664e4cdb3a9300a04"
server_key_version: "encryptionkey@0"
I20260925 19:07:23.505522 14230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:07:23.506297 14245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:23.506757 14230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:07:23.506856 14230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-2/wal
uuid: "55147c62bbda4e6da02eaa96a928c04e"
format_stamp: "Formatted at 2026-09-25 19:07:23 on dist-test-slave-rn98"
server_key: "23f60bef5f3a451db0e67f0a21243873"
server_key_iv: "6659e743102220c664e4cdb3a9300a04"
server_key_version: "encryptionkey@0"
I20260925 19:07:23.506933 14230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:07:23.515271 14230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:07:23.517681 14230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:07:23.517841 14230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:07:23.522306 14230 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.252:38513
I20260925 19:07:23.522384 14297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.252:38513 every 8 connection(s)
I20260925 19:07:23.522809 14230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-2/data/info.pb
I20260925 19:07:23.523774 14298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:07:23.525470 14298 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37375 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:07:23.529740 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 14230
I20260925 19:07:23.529824 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790363087086945-787-0/minicluster-data/master-2/wal/instance
I20260925 19:07:23.530027 787 external_mini_cluster.cc:1428] Setting key 09dc21c575106f379acc55200b0e1259
Sep 25 19:07:23 dist-test-slave-rn98 krb5kdc[14067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.252@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:07:23.535893 14151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.526304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59214 (local address 127.0.196.254:37375)
0925 19:07:23.526400 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:23.526402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:23.526692 (+ 290us) server_negotiation.cc:407] Connection header received
0925 19:07:23.527251 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:23.527253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:23.527265 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:23.527334 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:23.528014 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.528485 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.529581 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.529808 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.529876 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:23.532398 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:23.532400 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:23.532400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:23.532408 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:23.533995 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.534666 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.534668 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.534668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.534732 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.535038 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.535039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.535040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.535234 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:23.535323 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:23.535702 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 19:07:23.535805 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 19:07:23.536094 14300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.526213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:37375 (local address 127.0.0.1:59214)
0925 19:07:23.526478 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:23.526521 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:23.527129 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:23.527416 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:23.527419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:23.527696 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:07:23.527908 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:23.527913 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.528599 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.528600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:23.529471 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:23.529474 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.529906 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.529907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:23.530095 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:23.530727 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:07:23.530735 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:07:23.532293 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:07:23.534092 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:23.534098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:23.534126 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:23.534459 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:23.534829 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:23.534830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:23.534830 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:23.534950 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:23.535330 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:07:23.535336 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:07:23.535530 (+ 194us) client_negotiation.cc:769] Sending connection context
0925 19:07:23.535642 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 19:07:23.535839 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":131,"threads_started":1}
Sep 25 19:07:23 dist-test-slave-rn98 krb5kdc[14067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.252@KRBTEST.COM
I20260925 19:07:23.536739 14298 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.253" port: 37255 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:07:23.540737 14302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.530442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48612 (local address 127.0.196.252:38513)
0925 19:07:23.530646 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:23.530651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:23.530788 (+ 137us) server_negotiation.cc:407] Connection header received
0925 19:07:23.530823 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:23.530825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:23.530848 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:23.530940 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:23.531605 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.532381 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.533426 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.533779 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.533843 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:23.536958 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:23.536966 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:23.536966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:23.536974 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:23.539031 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.539433 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.539437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.539437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.539502 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.539727 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.539728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.539729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.539888 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:23.540067 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:23.540516 (+ 449us) server_negotiation.cc:299] Negotiation successful
0925 19:07:23.540629 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":105,"threads_started":1}
Sep 25 19:07:23 dist-test-slave-rn98 krb5kdc[14067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.252@KRBTEST.COM for kudu/127.0.196.253@KRBTEST.COM
I20260925 19:07:23.544373 14227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.537161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54914 (local address 127.0.196.253:37255)
0925 19:07:23.537239 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:23.537241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:23.537252 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:07:23.537303 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:23.537304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:23.537320 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:23.537369 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:23.537718 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.538596 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.539452 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.539648 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.539732 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:23.541768 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:23.541770 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:23.541770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:23.541780 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:23.543244 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.543594 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.543595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.543596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.543660 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.543855 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.543857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.543857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.543974 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:23.544050 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:23.544210 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 19:07:23.544281 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 19:07:23.544373 14300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.537074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:37255 (local address 127.0.0.1:54914)
0925 19:07:23.537175 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:23.537190 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:23.537238 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:23.537391 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:23.537392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:23.537561 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:07:23.537623 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:23.537625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.538715 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.538716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:23.539353 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:23.539355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.539732 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.539732 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:23.539807 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:23.540245 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:07:23.540249 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:07:23.541670 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:07:23.543350 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:23.543351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:23.543352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:23.543520 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:23.543727 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:23.543727 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:23.543728 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:23.543783 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:23.544042 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:07:23.544043 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:07:23.544087 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 19:07:23.544170 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 19:07:23.544270 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 19:07:23.544512 14148 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.252" port: 38513 } attempt: 2
I20260925 19:07:23.545053 14298 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.252" port: 38513 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 19:07:23 dist-test-slave-rn98 krb5kdc[14067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
Sep 25 19:07:23 dist-test-slave-rn98 krb5kdc[14067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for kudu/127.0.196.252@KRBTEST.COM
Sep 25 19:07:23 dist-test-slave-rn98 krb5kdc[14067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.252@KRBTEST.COM for kudu/127.0.196.252@KRBTEST.COM
I20260925 19:07:23.550791 14151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.541598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59218 (local address 127.0.196.254:37375)
0925 19:07:23.541645 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:23.541647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:23.541758 (+ 111us) server_negotiation.cc:407] Connection header received
0925 19:07:23.541851 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:23.541852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:23.541862 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:23.541903 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:23.542606 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.543150 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.543895 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.544129 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.544210 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:23.546970 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:23.546972 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:23.546972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:23.546982 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:23.548912 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.549343 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.549345 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.549345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.549398 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.549634 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.549634 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.549635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.549770 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:23.550552 (+ 782us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:23.550599 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 19:07:23.550714 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:07:23.553575 14150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.544798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:38513 (local address 127.0.0.1:48620)
0925 19:07:23.544843 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:23.544854 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:23.544893 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:23.545110 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:23.545111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:23.545285 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:07:23.545329 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:23.545330 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.546131 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.546132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:23.546889 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:23.546891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.547486 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.547487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:23.547555 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:23.547936 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:07:23.547941 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:07:23.549539 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:07:23.552291 (+ 2752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:23.552292 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:23.552293 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:23.552508 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:23.552741 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:23.552742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:23.552742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:23.552817 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:23.553178 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:07:23.553179 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:07:23.553233 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 19:07:23.553318 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 19:07:23.553422 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 19:07:23.553592 14302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.544907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48620 (local address 127.0.196.252:38513)
0925 19:07:23.544952 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:23.544954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:23.544962 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:07:23.544985 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:23.544986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:23.544999 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:23.545100 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:23.545400 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.546043 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.547004 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.547365 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.547704 (+ 339us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:23.549640 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:23.549642 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:23.549642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:23.549648 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:23.552196 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.552602 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.552604 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.552605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.552661 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.552897 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.552898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.552898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.553099 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:23.553166 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:23.553355 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 19:07:23.553480 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":22}
I20260925 19:07:23.554361 14300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.545284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:38513 (local address 127.0.0.1:48628)
0925 19:07:23.545464 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:23.545473 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:23.545506 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:23.545991 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:23.545992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:23.546156 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:07:23.546189 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:23.546191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.547125 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.547126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:23.547807 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:23.547809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.548447 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.548449 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:23.548523 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:23.549028 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:07:23.549033 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:07:23.550885 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:07:23.553211 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:23.553212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:23.553212 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:23.553465 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:23.553680 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:23.553681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:23.553681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:23.553744 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:23.554021 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:07:23.554022 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:07:23.554076 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 19:07:23.554158 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 19:07:23.554272 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260925 19:07:23.554733 14303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.545352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48628 (local address 127.0.196.252:38513)
0925 19:07:23.545864 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:23.545866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:23.545873 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:07:23.545890 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:23.545890 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:23.545898 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:23.545990 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:23.546278 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.547051 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.547874 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.548345 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.548427 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:23.551155 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:23.551157 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:23.551157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:23.551164 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:23.553111 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.553539 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.553540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.553541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.553608 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.553817 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.553818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.553818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.553942 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:23.554018 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:23.554191 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 19:07:23.554588 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":313,"server-negotiator.queue_time_us":494,"thread_start_us":129,"threads_started":1}
I20260925 19:07:23.555941 14298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55147c62bbda4e6da02eaa96a928c04e: Bootstrap starting.
I20260925 19:07:23.556178 14148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049: Bootstrap starting.
I20260925 19:07:23.556756 14148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049: Neither blocks nor log segments found. Creating new log.
I20260925 19:07:23.556800 14298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55147c62bbda4e6da02eaa96a928c04e: Neither blocks nor log segments found. Creating new log.
Sep 25 19:07:23 dist-test-slave-rn98 krb5kdc[14067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.253@KRBTEST.COM
I20260925 19:07:23.557001 14148 log.cc:824] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049: Log is configured to *not* fsync() on all Append() calls
I20260925 19:07:23.557137 14298 log.cc:824] T 00000000000000000000000000000000 P 55147c62bbda4e6da02eaa96a928c04e: Log is configured to *not* fsync() on all Append() calls
I20260925 19:07:23.557953 14148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049: No bootstrap required, opened a new log
I20260925 19:07:23.558157 14298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55147c62bbda4e6da02eaa96a928c04e: No bootstrap required, opened a new log
I20260925 19:07:23.559656 14298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55147c62bbda4e6da02eaa96a928c04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674f387cddb3491dbfdaa8a595cd0049" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37375 } } peers { permanent_uuid: "92dad4d16cc1493a9c1d27b1e6941c5c" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 37255 } } peers { permanent_uuid: "55147c62bbda4e6da02eaa96a928c04e" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 38513 } }
I20260925 19:07:23.559789 14298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55147c62bbda4e6da02eaa96a928c04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:07:23.559818 14298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55147c62bbda4e6da02eaa96a928c04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55147c62bbda4e6da02eaa96a928c04e, State: Initialized, Role: FOLLOWER
I20260925 19:07:23.559718 14148 raft_consensus.cc:348] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674f387cddb3491dbfdaa8a595cd0049" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37375 } } peers { permanent_uuid: "92dad4d16cc1493a9c1d27b1e6941c5c" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 37255 } } peers { permanent_uuid: "55147c62bbda4e6da02eaa96a928c04e" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 38513 } }
I20260925 19:07:23.559885 14148 raft_consensus.cc:374] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:07:23.559934 14148 raft_consensus.cc:729] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 674f387cddb3491dbfdaa8a595cd0049, State: Initialized, Role: FOLLOWER
I20260925 19:07:23.559954 14298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55147c62bbda4e6da02eaa96a928c04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674f387cddb3491dbfdaa8a595cd0049" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37375 } } peers { permanent_uuid: "92dad4d16cc1493a9c1d27b1e6941c5c" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 37255 } } peers { permanent_uuid: "55147c62bbda4e6da02eaa96a928c04e" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 38513 } }
I20260925 19:07:23.560075 14148 consensus_queue.cc:260] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674f387cddb3491dbfdaa8a595cd0049" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37375 } } peers { permanent_uuid: "92dad4d16cc1493a9c1d27b1e6941c5c" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 37255 } } peers { permanent_uuid: "55147c62bbda4e6da02eaa96a928c04e" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 38513 } }
I20260925 19:07:23.560230 14306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55147c62bbda4e6da02eaa96a928c04e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674f387cddb3491dbfdaa8a595cd0049" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37375 } } peers { permanent_uuid: "92dad4d16cc1493a9c1d27b1e6941c5c" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 37255 } } peers { permanent_uuid: "55147c62bbda4e6da02eaa96a928c04e" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 38513 } } }
I20260925 19:07:23.560309 14298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55147c62bbda4e6da02eaa96a928c04e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:07:23.560315 14306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55147c62bbda4e6da02eaa96a928c04e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:07:23.560411 14148 sys_catalog.cc:564] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:07:23.560520 14307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674f387cddb3491dbfdaa8a595cd0049" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37375 } } peers { permanent_uuid: "92dad4d16cc1493a9c1d27b1e6941c5c" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 37255 } } peers { permanent_uuid: "55147c62bbda4e6da02eaa96a928c04e" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 38513 } } }
I20260925 19:07:23.560680 14307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 19:07:23.562556 14323 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 55147c62bbda4e6da02eaa96a928c04e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:07:23.562711 14323 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260925 19:07:23.563328 14329 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:07:23.563385 14329 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:07:23.564888 14227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.551748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54916 (local address 127.0.196.253:37255)
0925 19:07:23.551818 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:23.551821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:23.551830 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:07:23.551854 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:23.551855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:23.551866 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:23.551930 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:23.552630 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.553338 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.554275 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.554557 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.554641 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:23.557440 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:23.557441 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:23.557442 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:23.557450 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:23.563135 (+ 5685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.563639 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.563640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.563640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.563704 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.563915 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.563916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.563916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.564050 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:23.564667 (+ 617us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:23.564712 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 19:07:23.564794 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 19:07:23.563742 14230 master_runner.cc:186] Error getting master registration for 127.0.196.253:37255: OK, instance_id { permanent_uuid: "92dad4d16cc1493a9c1d27b1e6941c5c" instance_seqno: 1790363243357403 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 19:07:23.566710 14230 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 19:07:23.564088 14080 master_runner.cc:186] Error getting master registration for 127.0.196.253:37255: OK, instance_id { permanent_uuid: "92dad4d16cc1493a9c1d27b1e6941c5c" instance_seqno: 1790363243357403 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 19:07:23.566911 14080 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 19:07:23.567404 14307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:07:23.567482 14307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674f387cddb3491dbfdaa8a595cd0049" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37375 } } peers { permanent_uuid: "92dad4d16cc1493a9c1d27b1e6941c5c" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 37255 } } peers { permanent_uuid: "55147c62bbda4e6da02eaa96a928c04e" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 38513 } }
I20260925 19:07:23.567708 14307 leader_election.cc:288] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92dad4d16cc1493a9c1d27b1e6941c5c (127.0.196.253:37255), 55147c62bbda4e6da02eaa96a928c04e (127.0.196.252:38513)
I20260925 19:07:23.568328 14272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "674f387cddb3491dbfdaa8a595cd0049" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55147c62bbda4e6da02eaa96a928c04e" is_pre_election: true
I20260925 19:07:23.568490 14272 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 55147c62bbda4e6da02eaa96a928c04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 674f387cddb3491dbfdaa8a595cd0049 in term 0.
I20260925 19:07:23.569097 787 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:07:23.569135 14099 leader_election.cc:302] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55147c62bbda4e6da02eaa96a928c04e, 674f387cddb3491dbfdaa8a595cd0049; no voters:
I20260925 19:07:23.569360 14307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:07:23.569423 14307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:07:23.569450 14307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:07:23.570427 14307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674f387cddb3491dbfdaa8a595cd0049" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37375 } } peers { permanent_uuid: "92dad4d16cc1493a9c1d27b1e6941c5c" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 37255 } } peers { permanent_uuid: "55147c62bbda4e6da02eaa96a928c04e" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 38513 } }
I20260925 19:07:23.570571 14307 leader_election.cc:288] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049 [CANDIDATE]: Term 1 election: Requested vote from peers 92dad4d16cc1493a9c1d27b1e6941c5c (127.0.196.253:37255), 55147c62bbda4e6da02eaa96a928c04e (127.0.196.252:38513)
I20260925 19:07:23.571043 14272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "674f387cddb3491dbfdaa8a595cd0049" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55147c62bbda4e6da02eaa96a928c04e"
I20260925 19:07:23.571130 14272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55147c62bbda4e6da02eaa96a928c04e [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:07:23.572098 14272 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 55147c62bbda4e6da02eaa96a928c04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 674f387cddb3491dbfdaa8a595cd0049 in term 1.
I20260925 19:07:23.572335 14099 leader_election.cc:302] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55147c62bbda4e6da02eaa96a928c04e, 674f387cddb3491dbfdaa8a595cd0049; no voters:
I20260925 19:07:23.572582 14307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:07:23.572759 14307 raft_consensus.cc:686] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049 [term 1 LEADER]: Becoming Leader. State: Replica: 674f387cddb3491dbfdaa8a595cd0049, State: Running, Role: LEADER
I20260925 19:07:23.572872 14307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674f387cddb3491dbfdaa8a595cd0049" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37375 } } peers { permanent_uuid: "92dad4d16cc1493a9c1d27b1e6941c5c" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 37255 } } peers { permanent_uuid: "55147c62bbda4e6da02eaa96a928c04e" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 38513 } }
I20260925 19:07:23.573350 14336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 674f387cddb3491dbfdaa8a595cd0049. Latest consensus state: current_term: 1 leader_uuid: "674f387cddb3491dbfdaa8a595cd0049" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674f387cddb3491dbfdaa8a595cd0049" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37375 } } peers { permanent_uuid: "92dad4d16cc1493a9c1d27b1e6941c5c" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 37255 } } peers { permanent_uuid: "55147c62bbda4e6da02eaa96a928c04e" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 38513 } } }
I20260925 19:07:23.573561 14336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049 [sys.catalog]: This master's current role is: LEADER
I20260925 19:07:23.574839 14338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:07:23.575150 14338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:07:23.585682 14227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.574414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54918 (local address 127.0.196.253:37255)
0925 19:07:23.574461 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:23.574463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:23.574929 (+ 466us) server_negotiation.cc:407] Connection header received
0925 19:07:23.575104 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:23.575119 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:23.575132 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:23.575165 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:23.576480 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.577323 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.578480 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.578898 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.578993 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:23.581024 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:23.581026 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:23.581026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:23.581035 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:23.582774 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.583471 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.583473 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.583473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.583602 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.583943 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.583944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.583944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.584172 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:23.584303 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:23.585306 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0925 19:07:23.585482 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:07:23.588864 14272 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 55147c62bbda4e6da02eaa96a928c04e [term 1 FOLLOWER]: Refusing update from remote peer 674f387cddb3491dbfdaa8a595cd0049: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:07:23.589670 14336 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55147c62bbda4e6da02eaa96a928c04e" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 38513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:07:23.591491 14306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55147c62bbda4e6da02eaa96a928c04e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 674f387cddb3491dbfdaa8a595cd0049. Latest consensus state: current_term: 1 leader_uuid: "674f387cddb3491dbfdaa8a595cd0049" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674f387cddb3491dbfdaa8a595cd0049" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37375 } } peers { permanent_uuid: "92dad4d16cc1493a9c1d27b1e6941c5c" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 37255 } } peers { permanent_uuid: "55147c62bbda4e6da02eaa96a928c04e" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 38513 } } }
I20260925 19:07:23.591635 14306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55147c62bbda4e6da02eaa96a928c04e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:07:23.591958 14303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.579201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48634 (local address 127.0.196.252:38513)
0925 19:07:23.579266 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:23.579268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:23.579501 (+ 233us) server_negotiation.cc:407] Connection header received
0925 19:07:23.580442 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:23.580444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:23.580478 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:23.580512 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:23.581190 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.582031 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.584219 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.584567 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.584637 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:23.586703 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:23.586705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:23.586705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:23.586714 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:23.588657 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.590403 (+ 1746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.590404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.590405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.590479 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.590807 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.590808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.590808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.590960 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:23.591056 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:23.591370 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 19:07:23.591465 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 19:07:23.593235 14151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.578922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59234 (local address 127.0.196.254:37375)
0925 19:07:23.578970 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:23.578972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:23.579306 (+ 334us) server_negotiation.cc:407] Connection header received
0925 19:07:23.579443 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:23.579444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:23.579460 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:23.579518 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:23.580354 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.581212 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.583180 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.583493 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.583562 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:23.587735 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:23.587736 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:23.587737 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:23.587745 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:23.589970 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.592195 (+ 2225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.592197 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.592197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.592265 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.592492 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.592493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.592493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.592653 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:23.592764 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:23.593058 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 19:07:23.593142 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:07:23.594069 14223 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.252" port: 38513 } attempt: 3
Sep 25 19:07:23 dist-test-slave-rn98 krb5kdc[14067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.253@KRBTEST.COM for kudu/127.0.196.252@KRBTEST.COM
I20260925 19:07:23.604213 14336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "674f387cddb3491dbfdaa8a595cd0049" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674f387cddb3491dbfdaa8a595cd0049" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37375 } } peers { permanent_uuid: "92dad4d16cc1493a9c1d27b1e6941c5c" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 37255 } } peers { permanent_uuid: "55147c62bbda4e6da02eaa96a928c04e" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 38513 } } }
I20260925 19:07:23.604331 14336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049 [sys.catalog]: This master's current role is: LEADER
I20260925 19:07:23.604921 14338 catalog_manager.cc:1347] Generated new cluster ID: 03418d1439bc4fa3b9fca1180324991e
I20260925 19:07:23.605000 14338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:07:23.605091 14306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55147c62bbda4e6da02eaa96a928c04e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "674f387cddb3491dbfdaa8a595cd0049" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674f387cddb3491dbfdaa8a595cd0049" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37375 } } peers { permanent_uuid: "92dad4d16cc1493a9c1d27b1e6941c5c" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 37255 } } peers { permanent_uuid: "55147c62bbda4e6da02eaa96a928c04e" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 38513 } } }
I20260925 19:07:23.605176 14306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55147c62bbda4e6da02eaa96a928c04e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:07:23.605264 14340 mvcc.cc:205] Tried to move back new op lower bound from 7333327845737029632 to 7333327845675114496. Current Snapshot: MvccSnapshot[applied={T|T < 7333327845737029632 or (T in {7333327845737029632})}]
I20260925 19:07:23.605331 14225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.594317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:38513 (local address 127.0.0.1:48650)
0925 19:07:23.594365 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:23.594376 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:23.594410 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:23.594594 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:23.594595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:23.594784 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:07:23.594835 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:23.594838 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.595801 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.595803 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:23.596614 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:23.596616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.597553 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.597554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:23.597615 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:23.598164 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:07:23.598171 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:07:23.600021 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:07:23.603560 (+ 3539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:23.603562 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:23.603563 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:23.603816 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:23.604240 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:23.604241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:23.604242 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:23.604320 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:23.604987 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:07:23.604988 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:07:23.605042 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 19:07:23.605094 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 19:07:23.605228 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 19:07:23.605365 14343 mvcc.cc:205] Tried to move back new op lower bound from 7333327845737029632 to 7333327845675114496. Current Snapshot: MvccSnapshot[applied={T|T < 7333327845737029632}]
I20260925 19:07:23.606282 14303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:23.594376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48650 (local address 127.0.196.252:38513)
0925 19:07:23.594430 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:23.594432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:23.594443 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:07:23.594467 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:23.594468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:23.594497 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:23.594578 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:23.594937 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.595699 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.597124 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:23.597432 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:23.597494 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:23.600162 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:23.600164 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:23.600164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:23.600209 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:23.603398 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.603960 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.603961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.603961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.604055 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:23.604562 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:23.604563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:23.604563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:23.604852 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:23.605875 (+ 1023us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:23.605952 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 19:07:23.606136 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:07:23.608135 14223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92dad4d16cc1493a9c1d27b1e6941c5c: Bootstrap starting.
I20260925 19:07:23.608953 14223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92dad4d16cc1493a9c1d27b1e6941c5c: Neither blocks nor log segments found. Creating new log.
I20260925 19:07:23.609308 14223 log.cc:824] T 00000000000000000000000000000000 P 92dad4d16cc1493a9c1d27b1e6941c5c: Log is configured to *not* fsync() on all Append() calls
I20260925 19:07:23.610452 14223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92dad4d16cc1493a9c1d27b1e6941c5c: No bootstrap required, opened a new log
I20260925 19:07:23.614418 14223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92dad4d16cc1493a9c1d27b1e6941c5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674f387cddb3491dbfdaa8a595cd0049" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37375 } } peers { permanent_uuid: "92dad4d16cc1493a9c1d27b1e6941c5c" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 37255 } } peers { permanent_uuid: "55147c62bbda4e6da02eaa96a928c04e" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 38513 } }
I20260925 19:07:23.614928 14223 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92dad4d16cc1493a9c1d27b1e6941c5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:07:23.615077 14223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92dad4d16cc1493a9c1d27b1e6941c5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92dad4d16cc1493a9c1d27b1e6941c5c, State: Initialized, Role: FOLLOWER
I20260925 19:07:23.615429 14223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92dad4d16cc1493a9c1d27b1e6941c5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674f387cddb3491dbfdaa8a595cd0049" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37375 } } peers { permanent_uuid: "92dad4d16cc1493a9c1d27b1e6941c5c" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 37255 } } peers { permanent_uuid: "55147c62bbda4e6da02eaa96a928c04e" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 38513 } }
I20260925 19:07:23.616214 14196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "674f387cddb3491dbfdaa8a595cd0049" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92dad4d16cc1493a9c1d27b1e6941c5c"
I20260925 19:07:23.616285 14195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92dad4d16cc1493a9c1d27b1e6941c5c [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:07:23.616225 14197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "674f387cddb3491dbfdaa8a595cd0049" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92dad4d16cc1493a9c1d27b1e6941c5c" is_pre_election: true
I20260925 19:07:23.617628 14195 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 92dad4d16cc1493a9c1d27b1e6941c5c [term 1 FOLLOWER]: Refusing update from remote peer 674f387cddb3491dbfdaa8a595cd0049: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:07:23.616165 14223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92dad4d16cc1493a9c1d27b1e6941c5c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:07:23.617897 14307 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92dad4d16cc1493a9c1d27b1e6941c5c" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 37255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:07:23.620046 14362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92dad4d16cc1493a9c1d27b1e6941c5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "674f387cddb3491dbfdaa8a595cd0049" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674f387cddb3491dbfdaa8a595cd0049" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37375 } } peers { permanent_uuid: "92dad4d16cc1493a9c1d27b1e6941c5c" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 37255 } } peers { permanent_uuid: "55147c62bbda4e6da02eaa96a928c04e" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 38513 } } }
I20260925 19:07:23.622840 14362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92dad4d16cc1493a9c1d27b1e6941c5c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:07:23.622984 14362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92dad4d16cc1493a9c1d27b1e6941c5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 674f387cddb3491dbfdaa8a595cd0049. Latest consensus state: current_term: 1 leader_uuid: "674f387cddb3491dbfdaa8a595cd0049" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674f387cddb3491dbfdaa8a595cd0049" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37375 } } peers { permanent_uuid: "92dad4d16cc1493a9c1d27b1e6941c5c" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 37255 } } peers { permanent_uuid: "55147c62bbda4e6da02eaa96a928c04e" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 38513 } } }
W20260925 19:07:23.623147 14363 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92dad4d16cc1493a9c1d27b1e6941c5c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:07:23.623231 14363 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:07:23.623145 14362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92dad4d16cc1493a9c1d27b1e6941c5c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:07:23.623335 14348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92dad4d16cc1493a9c1d27b1e6941c5c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "674f387cddb3491dbfdaa8a595cd0049" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674f387cddb3491dbfdaa8a595cd0049" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37375 } } peers { permanent_uuid: "92dad4d16cc1493a9c1d27b1e6941c5c" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 37255 } } peers { permanent_uuid: "55147c62bbda4e6da02eaa96a928c04e" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 38513 } } }
I20260925 19:07:23.626821 14348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92dad4d16cc1493a9c1d27b1e6941c5c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:07:23.627676 14357 mvcc.cc:205] Tried to move back new op lower bound from 7333327845737029632 to 7333327845675114496. Current Snapshot: MvccSnapshot[applied={T|T < 7333327845737029632}]
I20260925 19:07:23.637739 14307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "674f387cddb3491dbfdaa8a595cd0049" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674f387cddb3491dbfdaa8a595cd0049" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37375 } } peers { permanent_uuid: "92dad4d16cc1493a9c1d27b1e6941c5c" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 37255 } } peers { permanent_uuid: "55147c62bbda4e6da02eaa96a928c04e" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 38513 } } }
I20260925 19:07:23.637895 14307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049 [sys.catalog]: This master's current role is: LEADER
I20260925 19:07:23.642728 14338 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:07:23.643664 14338 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:07:23.653062 14338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 674f387cddb3491dbfdaa8a595cd0049: Generated new TSK 0
I20260925 19:07:23.653316 14338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:07:24.563210 14323 catalog_manager.cc:1259] Loaded cluster ID: 03418d1439bc4fa3b9fca1180324991e
I20260925 19:07:24.563288 14323 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 55147c62bbda4e6da02eaa96a928c04e: loading cluster ID for follower catalog manager: success
I20260925 19:07:24.564155 14323 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 55147c62bbda4e6da02eaa96a928c04e: acquiring CA information for follower catalog manager: success
I20260925 19:07:24.564477 14323 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 55147c62bbda4e6da02eaa96a928c04e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 19:07:24.623710 14363 catalog_manager.cc:1259] Loaded cluster ID: 03418d1439bc4fa3b9fca1180324991e
I20260925 19:07:24.623799 14363 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 92dad4d16cc1493a9c1d27b1e6941c5c: loading cluster ID for follower catalog manager: success
I20260925 19:07:24.624596 14363 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 92dad4d16cc1493a9c1d27b1e6941c5c: acquiring CA information for follower catalog manager: success
I20260925 19:07:24.624895 14363 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 92dad4d16cc1493a9c1d27b1e6941c5c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 19:07:24.706310 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 14080
I20260925 19:07:24.711689 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 14155
I20260925 19:07:24.716576 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 14230
2026-09-25T19:07:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2807 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:07:24 dist-test-slave-rn98 krb5kdc[14373](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:07:24 dist-test-slave-rn98 krb5kdc[14373](info): set up 2 sockets
Sep 25 19:07:24 dist-test-slave-rn98 krb5kdc[14373](info): commencing operation
krb5kdc: starting...
W20260925 19:07:25.770716 787 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:07:25 dist-test-slave-rn98 krb5kdc[14373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:07:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:07:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:07:25.899470 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:43773
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:34325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:43773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:07:25.982012 14386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:07:25.982234 14386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:07:25.982259 14386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:07:25.983494 14386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:07:25.983544 14386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:07:25.983559 14386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:07:25.983572 14386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:07:25.983584 14386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:07:25.985159 14386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:34325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:43773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:43773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:07:25.985438 14386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:07:25.985813 14386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:07:25.990821 14392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:25.990826 14391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:25.990841 14394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:25.991065 14386 server_base.cc:1034] running on GCE node
I20260925 19:07:25.991303 14386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:07:25.991640 14386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:07:25.992782 14386 hybrid_clock.cc:648] HybridClock initialized: now 1790363245992761 us; error 42 us; skew 500 ppm
Sep 25 19:07:25 dist-test-slave-rn98 krb5kdc[14373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:25.995472 14386 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:07:25.995947 14386 webserver.cc:460] Webserver started at http://127.0.196.254:36537/ using document root <none> and password file <none>
I20260925 19:07:25.996150 14386 fs_manager.cc:359] Metadata directory not provided
I20260925 19:07:25.996191 14386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:07:25.996277 14386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:07:25.997303 14386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "6d393d154cd64947a783afc711fb6112"
format_stamp: "Formatted at 2026-09-25 19:07:25 on dist-test-slave-rn98"
server_key: "0c91c77c46e6d9f0176a7683f0dc0c11"
server_key_iv: "2a3e36f70d38c4f891be37a3a6a35902"
server_key_version: "encryptionkey@0"
I20260925 19:07:25.997611 14386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "6d393d154cd64947a783afc711fb6112"
format_stamp: "Formatted at 2026-09-25 19:07:25 on dist-test-slave-rn98"
server_key: "0c91c77c46e6d9f0176a7683f0dc0c11"
server_key_iv: "2a3e36f70d38c4f891be37a3a6a35902"
server_key_version: "encryptionkey@0"
I20260925 19:07:25.998991 14386 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:07:25.999795 14401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:26.000056 14386 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:07:26.000173 14386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "6d393d154cd64947a783afc711fb6112"
format_stamp: "Formatted at 2026-09-25 19:07:25 on dist-test-slave-rn98"
server_key: "0c91c77c46e6d9f0176a7683f0dc0c11"
server_key_iv: "2a3e36f70d38c4f891be37a3a6a35902"
server_key_version: "encryptionkey@0"
I20260925 19:07:26.000254 14386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:07:26.020772 14386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:07:26.023300 14386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:07:26.023465 14386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:07:26.027817 14386 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:43773
I20260925 19:07:26.027889 14453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:43773 every 8 connection(s)
I20260925 19:07:26.028265 14386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:07:26.029181 14454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:07:26.031714 14454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d393d154cd64947a783afc711fb6112: Bootstrap starting.
I20260925 19:07:26.032323 14454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d393d154cd64947a783afc711fb6112: Neither blocks nor log segments found. Creating new log.
I20260925 19:07:26.032620 14454 log.cc:824] T 00000000000000000000000000000000 P 6d393d154cd64947a783afc711fb6112: Log is configured to *not* fsync() on all Append() calls
I20260925 19:07:26.033433 14454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d393d154cd64947a783afc711fb6112: No bootstrap required, opened a new log
I20260925 19:07:26.034979 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 14386
I20260925 19:07:26.035125 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:07:26.035346 787 external_mini_cluster.cc:1428] Setting key 26bbed566cccf3da3d405ca9daf6263b
I20260925 19:07:26.035358 14454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d393d154cd64947a783afc711fb6112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d393d154cd64947a783afc711fb6112" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 43773 } }
I20260925 19:07:26.035584 14454 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d393d154cd64947a783afc711fb6112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:07:26.035630 14454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d393d154cd64947a783afc711fb6112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d393d154cd64947a783afc711fb6112, State: Initialized, Role: FOLLOWER
I20260925 19:07:26.035792 14454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d393d154cd64947a783afc711fb6112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d393d154cd64947a783afc711fb6112" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 43773 } }
I20260925 19:07:26.035888 14454 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d393d154cd64947a783afc711fb6112 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:07:26.035926 14454 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d393d154cd64947a783afc711fb6112 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:07:26.035974 14454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d393d154cd64947a783afc711fb6112 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:07:26.036798 14454 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d393d154cd64947a783afc711fb6112 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d393d154cd64947a783afc711fb6112" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 43773 } }
I20260925 19:07:26.036916 14454 leader_election.cc:302] T 00000000000000000000000000000000 P 6d393d154cd64947a783afc711fb6112 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d393d154cd64947a783afc711fb6112; no voters:
I20260925 19:07:26.037084 14454 leader_election.cc:288] T 00000000000000000000000000000000 P 6d393d154cd64947a783afc711fb6112 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:07:26.037111 14459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d393d154cd64947a783afc711fb6112 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:07:26.037223 14459 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d393d154cd64947a783afc711fb6112 [term 1 LEADER]: Becoming Leader. State: Replica: 6d393d154cd64947a783afc711fb6112, State: Running, Role: LEADER
I20260925 19:07:26.037307 14459 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d393d154cd64947a783afc711fb6112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d393d154cd64947a783afc711fb6112" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 43773 } }
I20260925 19:07:26.037367 14454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d393d154cd64947a783afc711fb6112 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:07:26.037662 14461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d393d154cd64947a783afc711fb6112 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d393d154cd64947a783afc711fb6112" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d393d154cd64947a783afc711fb6112" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 43773 } } }
I20260925 19:07:26.037767 14461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d393d154cd64947a783afc711fb6112 [sys.catalog]: This master's current role is: LEADER
I20260925 19:07:26.037935 14460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d393d154cd64947a783afc711fb6112 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d393d154cd64947a783afc711fb6112. Latest consensus state: current_term: 1 leader_uuid: "6d393d154cd64947a783afc711fb6112" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d393d154cd64947a783afc711fb6112" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 43773 } } }
I20260925 19:07:26.037994 14460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d393d154cd64947a783afc711fb6112 [sys.catalog]: This master's current role is: LEADER
I20260925 19:07:26.038139 14464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:07:26.038929 14464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:07:26.040484 14464 catalog_manager.cc:1347] Generated new cluster ID: 34be5531477040a691bb1410ec5853cf
I20260925 19:07:26.040547 14464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:07:26 dist-test-slave-rn98 krb5kdc[14373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:07:26.047727 14464 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:07:26.048341 14464 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:07:26.049957 14458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:26.035927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56008 (local address 127.0.196.254:43773)
0925 19:07:26.036167 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:26.036178 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:26.036213 (+ 35us) server_negotiation.cc:407] Connection header received
0925 19:07:26.036665 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:26.036676 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:26.036747 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:26.036912 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:26.038474 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:26.039893 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:26.041046 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:26.041685 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:26.041816 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:26.044736 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:26.044741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:26.044742 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:26.044755 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:26.047740 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:26.048368 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:26.048373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:26.048373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:26.048565 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:26.048823 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:26.048824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:26.048824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:26.049188 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:26.049349 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:26.049513 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 19:07:26.049674 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":120,"threads_started":1}
I20260925 19:07:26.051144 787 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:07:26.057497 14464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d393d154cd64947a783afc711fb6112: Generated new TSK 0
I20260925 19:07:26.057777 14464 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:07:26.105458 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 14386
2026-09-25T19:07:26Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1386 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:07:26 dist-test-slave-rn98 krb5kdc[14481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:07:26 dist-test-slave-rn98 krb5kdc[14481](info): set up 2 sockets
Sep 25 19:07:26 dist-test-slave-rn98 krb5kdc[14481](info): commencing operation
krb5kdc: starting...
W20260925 19:07:27.163018 787 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:07:27 dist-test-slave-rn98 krb5kdc[14481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:07:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:07:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:07:27.291007 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:33023
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:39553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:33023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:07:27.373378 14494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:07:27.373658 14494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:07:27.373700 14494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:07:27.374963 14494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:07:27.375039 14494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:07:27.375063 14494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:07:27.375084 14494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:07:27.375114 14494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:07:27.376643 14494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:39553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:33023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:33023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:07:27.376946 14494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:07:27.377197 14494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:07:27.382331 14500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:27.382360 14499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:27.382606 14494 server_base.cc:1034] running on GCE node
W20260925 19:07:27.382339 14502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:27.383062 14494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:07:27.383443 14494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:07:27.384660 14494 hybrid_clock.cc:648] HybridClock initialized: now 1790363247384619 us; error 50 us; skew 500 ppm
Sep 25 19:07:27 dist-test-slave-rn98 krb5kdc[14481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:27.387737 14494 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:07:27.388270 14494 webserver.cc:460] Webserver started at http://127.0.196.254:37197/ using document root <none> and password file <none>
I20260925 19:07:27.388535 14494 fs_manager.cc:359] Metadata directory not provided
I20260925 19:07:27.388613 14494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:07:27.388779 14494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:07:27.389979 14494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "eadd875311e24529834eae46522bebcc"
format_stamp: "Formatted at 2026-09-25 19:07:27 on dist-test-slave-rn98"
server_key: "21f877945f38a77a0596af18d2e170e9"
server_key_iv: "e135c24a5820a29db39ed430c63757e7"
server_key_version: "encryptionkey@0"
I20260925 19:07:27.390406 14494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "eadd875311e24529834eae46522bebcc"
format_stamp: "Formatted at 2026-09-25 19:07:27 on dist-test-slave-rn98"
server_key: "21f877945f38a77a0596af18d2e170e9"
server_key_iv: "e135c24a5820a29db39ed430c63757e7"
server_key_version: "encryptionkey@0"
I20260925 19:07:27.392030 14494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:07:27.393046 14509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:27.393253 14494 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:07:27.393386 14494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "eadd875311e24529834eae46522bebcc"
format_stamp: "Formatted at 2026-09-25 19:07:27 on dist-test-slave-rn98"
server_key: "21f877945f38a77a0596af18d2e170e9"
server_key_iv: "e135c24a5820a29db39ed430c63757e7"
server_key_version: "encryptionkey@0"
I20260925 19:07:27.393548 14494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:07:27.427304 14494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:07:27.430137 14494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:07:27.430349 14494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:07:27.435163 14494 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:33023
I20260925 19:07:27.435201 14561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:33023 every 8 connection(s)
I20260925 19:07:27.435635 14494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:07:27.436193 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 14494
I20260925 19:07:27.436321 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:07:27.436583 787 external_mini_cluster.cc:1428] Setting key 0bd25dbe75128d502fbc8532f8cb5ac3
I20260925 19:07:27.437054 14562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:07:27.440661 14562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc: Bootstrap starting.
I20260925 19:07:27.441428 14562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc: Neither blocks nor log segments found. Creating new log.
I20260925 19:07:27.441764 14562 log.cc:824] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc: Log is configured to *not* fsync() on all Append() calls
I20260925 19:07:27.442606 14562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc: No bootstrap required, opened a new log
Sep 25 19:07:27 dist-test-slave-rn98 krb5kdc[14481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:07:27.445883 14562 raft_consensus.cc:348] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadd875311e24529834eae46522bebcc" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33023 } }
I20260925 19:07:27.446131 14562 raft_consensus.cc:374] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:07:27.446190 14562 raft_consensus.cc:729] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eadd875311e24529834eae46522bebcc, State: Initialized, Role: FOLLOWER
I20260925 19:07:27.446359 14562 consensus_queue.cc:260] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadd875311e24529834eae46522bebcc" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33023 } }
I20260925 19:07:27.446465 14562 raft_consensus.cc:388] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:07:27.446502 14562 raft_consensus.cc:482] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:07:27.446563 14562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:07:27.447464 14562 raft_consensus.cc:504] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadd875311e24529834eae46522bebcc" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33023 } }
I20260925 19:07:27.447592 14562 leader_election.cc:302] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eadd875311e24529834eae46522bebcc; no voters:
I20260925 19:07:27.447786 14562 leader_election.cc:288] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:07:27.447851 14567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:07:27.448276 14562 sys_catalog.cc:564] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:07:27.448441 14567 raft_consensus.cc:686] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [term 1 LEADER]: Becoming Leader. State: Replica: eadd875311e24529834eae46522bebcc, State: Running, Role: LEADER
I20260925 19:07:27.449855 14567 consensus_queue.cc:237] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadd875311e24529834eae46522bebcc" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33023 } }
I20260925 19:07:27.450416 14565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:27.437248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45640 (local address 127.0.196.254:33023)
0925 19:07:27.437525 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:27.437535 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:27.437565 (+ 30us) server_negotiation.cc:407] Connection header received
0925 19:07:27.438281 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:27.438290 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:27.438347 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:27.438486 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:27.439390 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:27.440343 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:27.441405 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:27.441967 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:27.442095 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:27.445625 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:27.445630 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:27.445631 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:27.445677 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:27.447970 (+ 2293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:27.448446 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:27.448450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:27.448450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:27.448569 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:27.448921 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:27.448923 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:27.448923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:27.449300 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:27.449937 (+ 637us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:27.450059 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 19:07:27.450213 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":103,"threads_started":1}
I20260925 19:07:27.450489 14568 sys_catalog.cc:455] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eadd875311e24529834eae46522bebcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadd875311e24529834eae46522bebcc" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33023 } } }
I20260925 19:07:27.450584 14568 sys_catalog.cc:458] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [sys.catalog]: This master's current role is: LEADER
I20260925 19:07:27.450735 14568 sys_catalog.cc:455] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader eadd875311e24529834eae46522bebcc. Latest consensus state: current_term: 1 leader_uuid: "eadd875311e24529834eae46522bebcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadd875311e24529834eae46522bebcc" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33023 } } }
I20260925 19:07:27.450794 14568 sys_catalog.cc:458] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [sys.catalog]: This master's current role is: LEADER
I20260925 19:07:27.451161 14581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:07:27.451681 14581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:07:27.459628 14581 catalog_manager.cc:1347] Generated new cluster ID: 3fc3175a784f48ecb81a0d8e457e7603
I20260925 19:07:27.459715 14581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:07:27.476279 14581 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:07:27.477224 14581 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:07:27.482064 14581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc: Generated new TSK 0
I20260925 19:07:27.482326 14581 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
I20260925 19:07:27.526949 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:0
--local_ip_for_outbound_sockets=127.0.196.193
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:33023
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:39553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:07:27.617341 14588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:07:27.617631 14588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:07:27.617662 14588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:07:27.619076 14588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:07:27.619125 14588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:07:27.619153 14588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:07:27.620700 14588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:39553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:33023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:07:27.620986 14588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:07:27.621239 14588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:07:27.626533 14593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:27.626519 14594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:27.626698 14588 server_base.cc:1034] running on GCE node
W20260925 19:07:27.626519 14596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:27.627031 14588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:07:27.627290 14588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:07:27.628463 14588 hybrid_clock.cc:648] HybridClock initialized: now 1790363247628447 us; error 48 us; skew 500 ppm
Sep 25 19:07:27 dist-test-slave-rn98 krb5kdc[14481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:27.631214 14588 init.cc:377] Logged in from keytab as kudu/127.0.196.193@KRBTEST.COM (short username kudu)
I20260925 19:07:27.631637 14588 webserver.cc:460] Webserver started at http://127.0.196.193:40425/ using document root <none> and password file <none>
I20260925 19:07:27.631836 14588 fs_manager.cc:359] Metadata directory not provided
I20260925 19:07:27.631877 14588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:07:27.631968 14588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:07:27.632992 14588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/data/instance:
uuid: "7c638a81e43148dc9b88b85642501874"
format_stamp: "Formatted at 2026-09-25 19:07:27 on dist-test-slave-rn98"
server_key: "8323b699a95b6ee25b7b8b60cc883d24"
server_key_iv: "be024b255af4912b7f144a4fb92b4f64"
server_key_version: "encryptionkey@0"
I20260925 19:07:27.633469 14588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/wal/instance:
uuid: "7c638a81e43148dc9b88b85642501874"
format_stamp: "Formatted at 2026-09-25 19:07:27 on dist-test-slave-rn98"
server_key: "8323b699a95b6ee25b7b8b60cc883d24"
server_key_iv: "be024b255af4912b7f144a4fb92b4f64"
server_key_version: "encryptionkey@0"
I20260925 19:07:27.634852 14588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 19:07:27.635632 14603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:27.635902 14588 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:07:27.636034 14588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "7c638a81e43148dc9b88b85642501874"
format_stamp: "Formatted at 2026-09-25 19:07:27 on dist-test-slave-rn98"
server_key: "8323b699a95b6ee25b7b8b60cc883d24"
server_key_iv: "be024b255af4912b7f144a4fb92b4f64"
server_key_version: "encryptionkey@0"
I20260925 19:07:27.636119 14588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:07:27.656513 14588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:07:27.659232 14588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:07:27.659435 14588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:07:27.659714 14588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:07:27.660065 14588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:07:27.660102 14588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:27.660151 14588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:07:27.660181 14588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:27.666587 14588 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:44299
I20260925 19:07:27.666693 14716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:44299 every 8 connection(s)
I20260925 19:07:27.667143 14588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
Sep 25 19:07:27 dist-test-slave-rn98 krb5kdc[14481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:07:27.674185 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 14588
I20260925 19:07:27.674299 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/wal/instance
I20260925 19:07:27.674553 787 external_mini_cluster.cc:1428] Setting key a9099cb3837144c87151a14ae6a2170e
I20260925 19:07:27.679607 14565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:27.667953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:34337 (local address 127.0.196.254:33023)
0925 19:07:27.668030 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:27.668032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:27.668253 (+ 221us) server_negotiation.cc:407] Connection header received
0925 19:07:27.668740 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:27.668741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:27.668752 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:27.668822 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:27.669912 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:27.670520 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:27.671649 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:27.671877 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:27.671960 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:27.674825 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:27.674827 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:27.674827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:27.674837 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:27.677926 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:27.678431 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:27.678433 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:27.678433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:27.678508 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:27.678847 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:27.678848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:27.678848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:27.678993 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:27.679080 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:27.679399 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 19:07:27.679500 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 19:07:27.679855 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:27.667923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.193:34337)
0925 19:07:27.668090 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:27.668133 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:27.668632 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:27.668898 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:27.668901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:27.669526 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:07:27.669789 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:27.669795 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:27.670709 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:27.670710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:27.671548 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:27.671550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:27.671971 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:27.671972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:27.672093 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:27.672696 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:07:27.672713 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:07:27.674706 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:07:27.678055 (+ 3349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:27.678065 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:27.678068 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:27.678340 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:27.678595 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:27.678595 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:27.678596 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:27.678769 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:27.679071 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:07:27.679078 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:07:27.679245 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 19:07:27.679363 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 19:07:27.679566 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
I20260925 19:07:27.680500 14717 heartbeater.cc:347] Connected to a master server at 127.0.196.254:33023
I20260925 19:07:27.680639 14717 heartbeater.cc:464] Registering TS with master...
I20260925 19:07:27.680847 14717 heartbeater.cc:511] Master 127.0.196.254:33023 requested a full tablet report, sending...
I20260925 19:07:27.681439 14526 ts_manager.cc:194] Registered new tserver with Master: 7c638a81e43148dc9b88b85642501874 (127.0.196.193:44299)
I20260925 19:07:27.682161 14526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.193@KRBTEST.COM'} at 127.0.196.193:34337
WARNING: no policy specified for kudu/127.0.196.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
I20260925 19:07:27.702347 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:0
--local_ip_for_outbound_sockets=127.0.196.194
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:33023
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:39553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:07:27.788242 14722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:07:27.788478 14722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:07:27.788511 14722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:07:27.789749 14722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:07:27.789801 14722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:07:27.789829 14722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:07:27.791440 14722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:39553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:33023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:07:27.791728 14722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:07:27.792007 14722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:07:27.797276 14727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:27.797276 14730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:27.797335 14728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:27.797863 14722 server_base.cc:1034] running on GCE node
I20260925 19:07:27.798075 14722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:07:27.798354 14722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:07:27.799522 14722 hybrid_clock.cc:648] HybridClock initialized: now 1790363247799513 us; error 46 us; skew 500 ppm
Sep 25 19:07:27 dist-test-slave-rn98 krb5kdc[14481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:27.802294 14722 init.cc:377] Logged in from keytab as kudu/127.0.196.194@KRBTEST.COM (short username kudu)
I20260925 19:07:27.802902 14722 webserver.cc:460] Webserver started at http://127.0.196.194:32849/ using document root <none> and password file <none>
I20260925 19:07:27.803170 14722 fs_manager.cc:359] Metadata directory not provided
I20260925 19:07:27.803243 14722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:07:27.803402 14722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:07:27.804553 14722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/data/instance:
uuid: "1eb37032863941f18f46907b9d7a07dc"
format_stamp: "Formatted at 2026-09-25 19:07:27 on dist-test-slave-rn98"
server_key: "48057dd7b0d0d70f9689ef5c7c3b2398"
server_key_iv: "81c872f7cffceb5bcc9a6dc2fa8e1f2d"
server_key_version: "encryptionkey@0"
I20260925 19:07:27.804943 14722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/wal/instance:
uuid: "1eb37032863941f18f46907b9d7a07dc"
format_stamp: "Formatted at 2026-09-25 19:07:27 on dist-test-slave-rn98"
server_key: "48057dd7b0d0d70f9689ef5c7c3b2398"
server_key_iv: "81c872f7cffceb5bcc9a6dc2fa8e1f2d"
server_key_version: "encryptionkey@0"
I20260925 19:07:27.806356 14722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:07:27.807340 14737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:27.807657 14722 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:07:27.807785 14722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "1eb37032863941f18f46907b9d7a07dc"
format_stamp: "Formatted at 2026-09-25 19:07:27 on dist-test-slave-rn98"
server_key: "48057dd7b0d0d70f9689ef5c7c3b2398"
server_key_iv: "81c872f7cffceb5bcc9a6dc2fa8e1f2d"
server_key_version: "encryptionkey@0"
I20260925 19:07:27.807868 14722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:07:27.820844 14722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:07:27.823305 14722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:07:27.823472 14722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:07:27.823740 14722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:07:27.824065 14722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:07:27.824100 14722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:27.824154 14722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:07:27.824191 14722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:27.830112 14722 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:42299
I20260925 19:07:27.830202 14850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:42299 every 8 connection(s)
I20260925 19:07:27.830524 14722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:07:27 dist-test-slave-rn98 krb5kdc[14481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:07:27.837679 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 14722
I20260925 19:07:27.837783 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/wal/instance
I20260925 19:07:27.837983 787 external_mini_cluster.cc:1428] Setting key 622f57fd9afafd25bca3c576561109b2
I20260925 19:07:27.841455 14565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:27.831352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:51041 (local address 127.0.196.254:33023)
0925 19:07:27.831409 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:27.831411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:27.831652 (+ 241us) server_negotiation.cc:407] Connection header received
0925 19:07:27.832271 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:27.832272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:27.832283 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:27.832349 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:27.833416 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:27.834292 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:27.835504 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:27.835703 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:27.835766 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:27.838179 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:27.838181 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:27.838181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:27.838191 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:27.839899 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:27.840411 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:27.840413 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:27.840413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:27.840483 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:27.840760 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:27.840761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:27.840762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:27.840891 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:27.840970 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:27.841273 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 19:07:27.841362 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:07:27.841678 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:27.831311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.194:51041)
0925 19:07:27.831482 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:27.831527 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:27.832090 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:27.832427 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:27.832431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:27.833045 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:07:27.833254 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:27.833259 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:27.834439 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:27.834440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:27.835418 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:27.835421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:27.835789 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:27.835789 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:27.835897 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:27.836400 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:07:27.836408 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:07:27.838044 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:07:27.839994 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:27.840004 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:27.840007 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:27.840311 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:27.840553 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:27.840554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:27.840555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:27.840689 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:27.840960 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:07:27.840967 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:07:27.841123 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 19:07:27.841245 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 19:07:27.841418 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":100,"threads_started":1}
I20260925 19:07:27.842319 14851 heartbeater.cc:347] Connected to a master server at 127.0.196.254:33023
I20260925 19:07:27.842435 14851 heartbeater.cc:464] Registering TS with master...
I20260925 19:07:27.842675 14851 heartbeater.cc:511] Master 127.0.196.254:33023 requested a full tablet report, sending...
I20260925 19:07:27.843107 14526 ts_manager.cc:194] Registered new tserver with Master: 1eb37032863941f18f46907b9d7a07dc (127.0.196.194:42299)
I20260925 19:07:27.843508 14526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.194@KRBTEST.COM'} at 127.0.196.194:51041
WARNING: no policy specified for kudu/127.0.196.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
I20260925 19:07:27.866460 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:0
--local_ip_for_outbound_sockets=127.0.196.195
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:33023
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:39553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:07:27.950203 14856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:07:27.950433 14856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:07:27.950462 14856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:07:27.951855 14856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:07:27.951908 14856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:07:27.951938 14856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:07:27.953598 14856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:39553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:33023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:07:27.953907 14856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:07:27.954185 14856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:07:27.959621 14862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:27.959722 14856 server_base.cc:1034] running on GCE node
W20260925 19:07:27.959672 14864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:27.959661 14861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:27.960074 14856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:07:27.960340 14856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:07:27.961567 14856 hybrid_clock.cc:648] HybridClock initialized: now 1790363247961554 us; error 47 us; skew 500 ppm
Sep 25 19:07:27 dist-test-slave-rn98 krb5kdc[14481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:27.964216 14856 init.cc:377] Logged in from keytab as kudu/127.0.196.195@KRBTEST.COM (short username kudu)
I20260925 19:07:27.964644 14856 webserver.cc:460] Webserver started at http://127.0.196.195:41353/ using document root <none> and password file <none>
I20260925 19:07:27.964857 14856 fs_manager.cc:359] Metadata directory not provided
I20260925 19:07:27.964898 14856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:07:27.964989 14856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:07:27.966027 14856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/data/instance:
uuid: "04f233d4c02b413daf423a661674cd48"
format_stamp: "Formatted at 2026-09-25 19:07:27 on dist-test-slave-rn98"
server_key: "8ac3f86a1be2cd8a1a4d3cc5c89d3de8"
server_key_iv: "2e1b15cb0a9a9a80b0bd66d5e9420906"
server_key_version: "encryptionkey@0"
I20260925 19:07:27.966331 14856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/wal/instance:
uuid: "04f233d4c02b413daf423a661674cd48"
format_stamp: "Formatted at 2026-09-25 19:07:27 on dist-test-slave-rn98"
server_key: "8ac3f86a1be2cd8a1a4d3cc5c89d3de8"
server_key_iv: "2e1b15cb0a9a9a80b0bd66d5e9420906"
server_key_version: "encryptionkey@0"
I20260925 19:07:27.967809 14856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:07:27.968602 14871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:27.968811 14856 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:07:27.968899 14856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "04f233d4c02b413daf423a661674cd48"
format_stamp: "Formatted at 2026-09-25 19:07:27 on dist-test-slave-rn98"
server_key: "8ac3f86a1be2cd8a1a4d3cc5c89d3de8"
server_key_iv: "2e1b15cb0a9a9a80b0bd66d5e9420906"
server_key_version: "encryptionkey@0"
I20260925 19:07:27.968952 14856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:07:27.976676 14856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:07:27.979017 14856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:07:27.979166 14856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:07:27.979413 14856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:07:27.979736 14856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:07:27.979772 14856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:27.979795 14856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:07:27.979807 14856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:27.985862 14856 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:44467
I20260925 19:07:27.985942 14984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:44467 every 8 connection(s)
I20260925 19:07:27.986285 14856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
I20260925 19:07:27.991494 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 14856
I20260925 19:07:27.991565 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/wal/instance
I20260925 19:07:27.991770 787 external_mini_cluster.cc:1428] Setting key a0e9d24031c8e7a0306716efe2b717c2
Sep 25 19:07:27 dist-test-slave-rn98 krb5kdc[14481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:07:27.996196 14565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:27.987212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:56953 (local address 127.0.196.254:33023)
0925 19:07:27.987312 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:27.987319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:27.987533 (+ 214us) server_negotiation.cc:407] Connection header received
0925 19:07:27.988080 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:27.988082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:27.988094 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:27.988147 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:27.989215 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:27.989726 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:27.990813 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:27.991012 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:27.991095 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:27.993606 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:27.993607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:27.993607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:27.993615 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:27.994912 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:27.995286 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:27.995287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:27.995287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:27.995342 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:27.995590 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:27.995590 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:27.995591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:27.995708 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:27.995764 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:27.996064 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 19:07:27.996128 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 19:07:27.996495 14987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:27.987139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.195:56953)
0925 19:07:27.987370 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:27.987416 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:27.987974 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:27.988234 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:27.988242 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:27.988858 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:07:27.989105 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:27.989114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:27.989822 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:27.989824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:27.990715 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:27.990717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:27.991105 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:27.991106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:27.991239 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:27.991827 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:07:27.991835 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:07:27.993503 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:07:27.994986 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:27.994991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:27.994993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:27.995216 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:27.995402 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:27.995403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:27.995403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:27.995520 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:27.995793 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:07:27.995799 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:07:27.995927 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 19:07:27.996054 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 19:07:27.996190 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":91,"threads_started":1}
I20260925 19:07:27.997045 14985 heartbeater.cc:347] Connected to a master server at 127.0.196.254:33023
I20260925 19:07:27.997159 14985 heartbeater.cc:464] Registering TS with master...
I20260925 19:07:27.997361 14985 heartbeater.cc:511] Master 127.0.196.254:33023 requested a full tablet report, sending...
I20260925 19:07:27.997751 14526 ts_manager.cc:194] Registered new tserver with Master: 04f233d4c02b413daf423a661674cd48 (127.0.196.195:44467)
I20260925 19:07:27.998147 14526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.195@KRBTEST.COM'} at 127.0.196.195:56953
I20260925 19:07:28.003789 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:07:28.011987 14565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:28.005200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45654 (local address 127.0.196.254:33023)
0925 19:07:28.005282 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:28.005285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:28.005396 (+ 111us) server_negotiation.cc:407] Connection header received
0925 19:07:28.005480 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:28.005481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:28.005491 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:28.005621 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:28.006221 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.006790 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.007578 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.007781 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.007865 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:28.009126 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:28.009127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:28.009128 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:28.009135 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:28.010700 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:28.011110 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:28.011111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:28.011111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:28.011177 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:28.011386 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:28.011387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:28.011388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:28.011553 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:28.011659 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:28.011783 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 19:07:28.011875 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 19:07:28.013833 14526 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:07:28.015594 14526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:07:28.022454 14996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:28.018198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:44299 (local address 127.0.0.1:42348)
0925 19:07:28.018508 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:28.018519 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:28.018582 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:28.018907 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:28.018910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:28.018913 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:07:28.018995 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:28.018997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.020037 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.020038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:28.021304 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:28.021307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.022108 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.022109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:28.022196 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:28.022210 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 19:07:28.022311 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 19:07:28.022372 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":9,"thread_start_us":81,"threads_started":1}
I20260925 19:07:28.022466 15000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:28.018417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42348 (local address 127.0.196.193:44299)
0925 19:07:28.018730 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:28.018734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:28.018743 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:07:28.018781 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:28.018783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:28.018797 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:28.019075 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:07:28.019162 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.019930 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.021403 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.022004 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.022098 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:28.022165 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:28.022331 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 19:07:28.022372 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":81,"threads_started":1}
I20260925 19:07:28.022737 14997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:28.018198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:42299 (local address 127.0.0.1:43420)
0925 19:07:28.018368 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:28.018409 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:28.018475 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:28.018921 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:28.018922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:28.018923 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:07:28.018995 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:28.018998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.020382 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.020383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:28.021420 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:28.021424 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.022451 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.022452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:28.022520 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:28.022529 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 19:07:28.022578 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 19:07:28.022655 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":10,"thread_start_us":60,"threads_started":1}
I20260925 19:07:28.022836 15001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:28.018600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50388 (local address 127.0.196.195:44467)
0925 19:07:28.019086 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:28.019091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:28.019103 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:07:28.019143 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:28.019145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:28.019160 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:28.019245 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:07:28.019374 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.020438 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.021629 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.022300 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.022359 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:28.022411 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:28.022701 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 19:07:28.022742 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":111,"threads_started":1}
I20260925 19:07:28.023005 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:28.018397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:44467 (local address 127.0.0.1:50388)
0925 19:07:28.018805 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:28.018815 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:28.018859 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:28.019265 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:28.019265 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:28.019267 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:07:28.019309 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:28.019310 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.020535 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.020537 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:28.021536 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:28.021539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.022449 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.022451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:28.022520 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:28.022530 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 19:07:28.022578 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 19:07:28.022654 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":5,"thread_start_us":65,"threads_started":1}
I20260925 19:07:28.023255 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:28.018324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43420 (local address 127.0.196.194:42299)
0925 19:07:28.018669 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:28.018675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:28.018686 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:07:28.018731 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:28.018734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:28.018752 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:28.018834 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:07:28.019085 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.020263 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.021514 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.022280 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.022357 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:28.022411 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:28.023034 (+ 623us) server_negotiation.cc:299] Negotiation successful
0925 19:07:28.023173 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":94,"threads_started":1}
I20260925 19:07:28.023747 14651 tablet_service.cc:1467] Processing CreateTablet for tablet 9bf8d0e6bca74c49a1f92891a16cd6df (DEFAULT_TABLE table=test-table [id=9f0da79eda654244b1a72cacb86bbbe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:07:28.023720 14919 tablet_service.cc:1467] Processing CreateTablet for tablet 9bf8d0e6bca74c49a1f92891a16cd6df (DEFAULT_TABLE table=test-table [id=9f0da79eda654244b1a72cacb86bbbe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:07:28.023820 14785 tablet_service.cc:1467] Processing CreateTablet for tablet 9bf8d0e6bca74c49a1f92891a16cd6df (DEFAULT_TABLE table=test-table [id=9f0da79eda654244b1a72cacb86bbbe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:07:28.024077 14919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bf8d0e6bca74c49a1f92891a16cd6df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:07:28.024106 14651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bf8d0e6bca74c49a1f92891a16cd6df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:07:28.024106 14785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bf8d0e6bca74c49a1f92891a16cd6df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:07:28.026732 15002 tablet_bootstrap.cc:492] T 9bf8d0e6bca74c49a1f92891a16cd6df P 04f233d4c02b413daf423a661674cd48: Bootstrap starting.
I20260925 19:07:28.027398 15002 tablet_bootstrap.cc:654] T 9bf8d0e6bca74c49a1f92891a16cd6df P 04f233d4c02b413daf423a661674cd48: Neither blocks nor log segments found. Creating new log.
I20260925 19:07:28.027465 15003 tablet_bootstrap.cc:492] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc: Bootstrap starting.
I20260925 19:07:28.027756 15004 tablet_bootstrap.cc:492] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874: Bootstrap starting.
I20260925 19:07:28.028003 15002 log.cc:824] T 9bf8d0e6bca74c49a1f92891a16cd6df P 04f233d4c02b413daf423a661674cd48: Log is configured to *not* fsync() on all Append() calls
I20260925 19:07:28.028162 15003 tablet_bootstrap.cc:654] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc: Neither blocks nor log segments found. Creating new log.
I20260925 19:07:28.028430 15004 tablet_bootstrap.cc:654] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874: Neither blocks nor log segments found. Creating new log.
I20260925 19:07:28.028738 15003 log.cc:824] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc: Log is configured to *not* fsync() on all Append() calls
I20260925 19:07:28.028843 15002 tablet_bootstrap.cc:492] T 9bf8d0e6bca74c49a1f92891a16cd6df P 04f233d4c02b413daf423a661674cd48: No bootstrap required, opened a new log
I20260925 19:07:28.028918 15002 ts_tablet_manager.cc:1397] T 9bf8d0e6bca74c49a1f92891a16cd6df P 04f233d4c02b413daf423a661674cd48: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:07:28.028995 15004 log.cc:824] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874: Log is configured to *not* fsync() on all Append() calls
I20260925 19:07:28.029534 15003 tablet_bootstrap.cc:492] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc: No bootstrap required, opened a new log
I20260925 19:07:28.029668 15003 ts_tablet_manager.cc:1397] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:07:28.029711 15004 tablet_bootstrap.cc:492] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874: No bootstrap required, opened a new log
I20260925 19:07:28.029768 15004 ts_tablet_manager.cc:1397] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:07:28.031055 15002 raft_consensus.cc:348] T 9bf8d0e6bca74c49a1f92891a16cd6df P 04f233d4c02b413daf423a661674cd48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } }
I20260925 19:07:28.031253 15002 raft_consensus.cc:374] T 9bf8d0e6bca74c49a1f92891a16cd6df P 04f233d4c02b413daf423a661674cd48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:07:28.031292 15002 raft_consensus.cc:729] T 9bf8d0e6bca74c49a1f92891a16cd6df P 04f233d4c02b413daf423a661674cd48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f233d4c02b413daf423a661674cd48, State: Initialized, Role: FOLLOWER
I20260925 19:07:28.031409 15002 consensus_queue.cc:260] T 9bf8d0e6bca74c49a1f92891a16cd6df P 04f233d4c02b413daf423a661674cd48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } }
I20260925 19:07:28.031332 15004 raft_consensus.cc:348] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } }
I20260925 19:07:28.031514 15004 raft_consensus.cc:374] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:07:28.031561 15004 raft_consensus.cc:729] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c638a81e43148dc9b88b85642501874, State: Initialized, Role: FOLLOWER
I20260925 19:07:28.031670 15004 consensus_queue.cc:260] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } }
I20260925 19:07:28.031723 15002 ts_tablet_manager.cc:1428] T 9bf8d0e6bca74c49a1f92891a16cd6df P 04f233d4c02b413daf423a661674cd48: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:07:28.031901 14985 heartbeater.cc:503] Master 127.0.196.254:33023 was elected leader, sending a full tablet report...
I20260925 19:07:28.031951 15004 ts_tablet_manager.cc:1428] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:07:28.031816 15003 raft_consensus.cc:348] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } }
I20260925 19:07:28.031985 15003 raft_consensus.cc:374] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:07:28.032021 15003 raft_consensus.cc:729] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb37032863941f18f46907b9d7a07dc, State: Initialized, Role: FOLLOWER
I20260925 19:07:28.032112 14717 heartbeater.cc:503] Master 127.0.196.254:33023 was elected leader, sending a full tablet report...
I20260925 19:07:28.032171 15003 consensus_queue.cc:260] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } }
I20260925 19:07:28.032474 15003 ts_tablet_manager.cc:1428] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 19:07:28.032503 14851 heartbeater.cc:503] Master 127.0.196.254:33023 was elected leader, sending a full tablet report...
W20260925 19:07:28.081199 14852 tablet.cc:2369] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:07:28.168056 14718 tablet.cc:2369] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:07:28.236975 14986 tablet.cc:2369] T 9bf8d0e6bca74c49a1f92891a16cd6df P 04f233d4c02b413daf423a661674cd48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:07:28.265137 15010 raft_consensus.cc:482] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:07:28.265275 15010 raft_consensus.cc:504] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } }
I20260925 19:07:28.265668 15010 leader_election.cc:288] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04f233d4c02b413daf423a661674cd48 (127.0.196.195:44467), 7c638a81e43148dc9b88b85642501874 (127.0.196.193:44299)
I20260925 19:07:28.268764 15001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:28.265985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:43211 (local address 127.0.196.195:44467)
0925 19:07:28.266029 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:28.266031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:28.266041 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:07:28.266114 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:28.266115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:28.266126 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:28.266199 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:07:28.266403 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.266970 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.267984 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.268383 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.268440 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:28.268447 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:28.268643 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 19:07:28.268685 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:07:28.268777 15000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:28.265959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:49627 (local address 127.0.196.193:44299)
0925 19:07:28.266008 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:28.266011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:28.266243 (+ 232us) server_negotiation.cc:407] Connection header received
0925 19:07:28.266282 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:28.266283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:28.266295 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:28.266353 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:07:28.266530 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.267101 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.268044 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.268395 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.268498 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:28.268506 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:28.268677 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 19:07:28.268716 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:07:28.268765 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:28.265884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:44467 (local address 127.0.196.194:43211)
0925 19:07:28.265939 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:28.265952 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:28.266036 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:28.266237 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:28.266238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:28.266240 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:07:28.266325 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:28.266327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.267059 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.267060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:28.267883 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:28.267885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.268471 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.268472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:28.268561 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:28.268565 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:07:28.268607 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 19:07:28.268672 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 19:07:28.268787 15011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:28.265887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:44299 (local address 127.0.196.194:49627)
0925 19:07:28.266154 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:28.266165 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:28.266225 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:28.266382 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:28.266383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:28.266385 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:07:28.266458 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:28.266460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.267189 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.267190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:28.267946 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:28.267949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.268516 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.268517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:28.268570 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:28.268573 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:07:28.268652 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 19:07:28.268688 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":66,"thread_start_us":116,"threads_started":1}
I20260925 19:07:28.269166 14939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bf8d0e6bca74c49a1f92891a16cd6df" candidate_uuid: "1eb37032863941f18f46907b9d7a07dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f233d4c02b413daf423a661674cd48" is_pre_election: true
I20260925 19:07:28.269349 14939 raft_consensus.cc:2454] T 9bf8d0e6bca74c49a1f92891a16cd6df P 04f233d4c02b413daf423a661674cd48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eb37032863941f18f46907b9d7a07dc in term 0.
I20260925 19:07:28.269435 14671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bf8d0e6bca74c49a1f92891a16cd6df" candidate_uuid: "1eb37032863941f18f46907b9d7a07dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c638a81e43148dc9b88b85642501874" is_pre_election: true
I20260925 19:07:28.269613 14671 raft_consensus.cc:2454] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eb37032863941f18f46907b9d7a07dc in term 0.
I20260925 19:07:28.269632 14738 leader_election.cc:302] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04f233d4c02b413daf423a661674cd48, 1eb37032863941f18f46907b9d7a07dc; no voters:
I20260925 19:07:28.269779 15010 raft_consensus.cc:2781] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:07:28.269827 15010 raft_consensus.cc:482] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:07:28.269845 15010 raft_consensus.cc:3037] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:07:28.270592 15010 raft_consensus.cc:504] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } }
I20260925 19:07:28.270821 15010 leader_election.cc:288] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc [CANDIDATE]: Term 1 election: Requested vote from peers 04f233d4c02b413daf423a661674cd48 (127.0.196.195:44467), 7c638a81e43148dc9b88b85642501874 (127.0.196.193:44299)
I20260925 19:07:28.271006 14939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bf8d0e6bca74c49a1f92891a16cd6df" candidate_uuid: "1eb37032863941f18f46907b9d7a07dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f233d4c02b413daf423a661674cd48"
I20260925 19:07:28.271018 14671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bf8d0e6bca74c49a1f92891a16cd6df" candidate_uuid: "1eb37032863941f18f46907b9d7a07dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c638a81e43148dc9b88b85642501874"
I20260925 19:07:28.271090 14939 raft_consensus.cc:3037] T 9bf8d0e6bca74c49a1f92891a16cd6df P 04f233d4c02b413daf423a661674cd48 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:07:28.271090 14671 raft_consensus.cc:3037] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:07:28.271986 14939 raft_consensus.cc:2454] T 9bf8d0e6bca74c49a1f92891a16cd6df P 04f233d4c02b413daf423a661674cd48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eb37032863941f18f46907b9d7a07dc in term 1.
I20260925 19:07:28.271986 14671 raft_consensus.cc:2454] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eb37032863941f18f46907b9d7a07dc in term 1.
I20260925 19:07:28.272125 14738 leader_election.cc:302] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04f233d4c02b413daf423a661674cd48, 1eb37032863941f18f46907b9d7a07dc; no voters:
I20260925 19:07:28.272204 15010 raft_consensus.cc:2781] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:07:28.272346 15010 raft_consensus.cc:686] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc [term 1 LEADER]: Becoming Leader. State: Replica: 1eb37032863941f18f46907b9d7a07dc, State: Running, Role: LEADER
I20260925 19:07:28.272452 15010 consensus_queue.cc:237] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } }
I20260925 19:07:28.273396 14526 catalog_manager.cc:5495] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eb37032863941f18f46907b9d7a07dc (127.0.196.194). New cstate: current_term: 1 leader_uuid: "1eb37032863941f18f46907b9d7a07dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } health_report { overall_health: UNKNOWN } } }
I20260925 19:07:28.310534 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:28.308124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43426 (local address 127.0.196.194:42299)
0925 19:07:28.308187 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:28.308189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:28.308197 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:07:28.308250 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:28.308251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:28.308268 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:28.308329 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:07:28.308459 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.308922 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.309756 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.309932 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.309981 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:28.310164 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:07:28.310302 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:07:28.310400 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:28.310419 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 19:07:28.310438 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 19:07:28.312579 14939 raft_consensus.cc:1261] T 9bf8d0e6bca74c49a1f92891a16cd6df P 04f233d4c02b413daf423a661674cd48 [term 1 FOLLOWER]: Refusing update from remote peer 1eb37032863941f18f46907b9d7a07dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:07:28.312579 14671 raft_consensus.cc:1261] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874 [term 1 FOLLOWER]: Refusing update from remote peer 1eb37032863941f18f46907b9d7a07dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:07:28.313040 15010 consensus_queue.cc:1035] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:07:28.313199 15012 consensus_queue.cc:1035] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:07:28.316924 15017 mvcc.cc:205] Tried to move back new op lower bound from 7333327865085267968 to 7333327864924827648. Current Snapshot: MvccSnapshot[applied={T|T < 7333327865085267968 or (T in {7333327865085267968})}]
I20260925 19:07:28.317075 15015 mvcc.cc:205] Tried to move back new op lower bound from 7333327865085267968 to 7333327864924827648. Current Snapshot: MvccSnapshot[applied={T|T < 7333327865085267968 or (T in {7333327865085267968})}]
I20260925 19:07:28.318645 14526 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:07:28.318763 14526 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:07:28.319604 14526 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc: Sending DeleteTablet for 3 replicas of tablet 9bf8d0e6bca74c49a1f92891a16cd6df
I20260925 19:07:28.319975 14651 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bf8d0e6bca74c49a1f92891a16cd6df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:07:28 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:42348
I20260925 19:07:28.320025 14919 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bf8d0e6bca74c49a1f92891a16cd6df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:07:28 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:50388
I20260925 19:07:28.320073 14785 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bf8d0e6bca74c49a1f92891a16cd6df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:07:28 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:43420
I20260925 19:07:28.320228 15025 tablet_replica.cc:330] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874: stopping tablet replica
I20260925 19:07:28.320248 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 14588
I20260925 19:07:28.320333 15025 raft_consensus.cc:2229] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:07:28.320447 15025 raft_consensus.cc:2258] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:07:28.320824 15026 tablet_replica.cc:330] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc: stopping tablet replica
I20260925 19:07:28.320914 15026 raft_consensus.cc:2229] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:07:28.321033 15026 raft_consensus.cc:2258] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:07:28.321699 15024 tablet_replica.cc:330] T 9bf8d0e6bca74c49a1f92891a16cd6df P 04f233d4c02b413daf423a661674cd48: stopping tablet replica
I20260925 19:07:28.321817 15024 raft_consensus.cc:2229] T 9bf8d0e6bca74c49a1f92891a16cd6df P 04f233d4c02b413daf423a661674cd48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:07:28.321924 15024 raft_consensus.cc:2258] T 9bf8d0e6bca74c49a1f92891a16cd6df P 04f233d4c02b413daf423a661674cd48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:07:28.322353 15024 ts_tablet_manager.cc:1905] T 9bf8d0e6bca74c49a1f92891a16cd6df P 04f233d4c02b413daf423a661674cd48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:07:28.323413 15026 ts_tablet_manager.cc:1905] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:07:28.324252 15024 ts_tablet_manager.cc:1918] T 9bf8d0e6bca74c49a1f92891a16cd6df P 04f233d4c02b413daf423a661674cd48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:07:28.324340 15024 log.cc:1196] T 9bf8d0e6bca74c49a1f92891a16cd6df P 04f233d4c02b413daf423a661674cd48: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/wal/wals/9bf8d0e6bca74c49a1f92891a16cd6df
I20260925 19:07:28.324704 15024 ts_tablet_manager.cc:1939] T 9bf8d0e6bca74c49a1f92891a16cd6df P 04f233d4c02b413daf423a661674cd48: Deleting consensus metadata
I20260925 19:07:28.325201 14510 catalog_manager.cc:4841] TS 04f233d4c02b413daf423a661674cd48 (127.0.196.195:44467): tablet 9bf8d0e6bca74c49a1f92891a16cd6df (table test-table [id=9f0da79eda654244b1a72cacb86bbbe3]) successfully deleted
I20260925 19:07:28.325717 15026 ts_tablet_manager.cc:1918] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:07:28.325816 15026 log.cc:1196] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/wal/wals/9bf8d0e6bca74c49a1f92891a16cd6df
I20260925 19:07:28.326066 15026 ts_tablet_manager.cc:1939] T 9bf8d0e6bca74c49a1f92891a16cd6df P 1eb37032863941f18f46907b9d7a07dc: Deleting consensus metadata
I20260925 19:07:28.326474 14510 catalog_manager.cc:4841] TS 1eb37032863941f18f46907b9d7a07dc (127.0.196.194:42299): tablet 9bf8d0e6bca74c49a1f92891a16cd6df (table test-table [id=9f0da79eda654244b1a72cacb86bbbe3]) successfully deleted
W20260925 19:07:28.327682 14513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.196.193:44299 (error 108)
I20260925 19:07:28.328078 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 14722
I20260925 19:07:28.329584 14997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:28.329395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:44299 (local address 127.0.0.1:42354)
0925 19:07:28.329482 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:28.329538 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.193:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 19:07:28.329723 14513 catalog_manager.cc:4584] TS 7c638a81e43148dc9b88b85642501874 (127.0.196.193:44299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9bf8d0e6bca74c49a1f92891a16cd6df: Network error: Client connection negotiation failed: client connection to 127.0.196.193:44299: connect: Connection refused (error 111)
I20260925 19:07:28.333808 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 14856
I20260925 19:07:28.338979 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 14494
I20260925 19:07:28.345086 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:33023
--webserver_interface=127.0.196.254
--webserver_port=37197
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:39553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:33023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:07:28.427795 15028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:07:28.428021 15028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:07:28.428045 15028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:07:28.429315 15028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:07:28.429365 15028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:07:28.429379 15028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:07:28.429392 15028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:07:28.429404 15028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:07:28.430967 15028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:39553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:33023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:33023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=37197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:07:28.431257 15028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:07:28.431505 15028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:07:28.440896 15035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:28.440948 15037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:28.440984 15034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:28.441386 15028 server_base.cc:1034] running on GCE node
I20260925 19:07:28.441672 15028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:07:28.442022 15028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:07:28.443197 15028 hybrid_clock.cc:648] HybridClock initialized: now 1790363248443195 us; error 63 us; skew 500 ppm
Sep 25 19:07:28 dist-test-slave-rn98 krb5kdc[14481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:28.452801 15028 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:07:28.453404 15028 webserver.cc:460] Webserver started at http://127.0.196.254:37197/ using document root <none> and password file <none>
I20260925 19:07:28.453688 15028 fs_manager.cc:359] Metadata directory not provided
I20260925 19:07:28.453783 15028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:07:28.455492 15028 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:07:28.456297 15044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:28.456529 15028 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:07:28.456660 15028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "eadd875311e24529834eae46522bebcc"
format_stamp: "Formatted at 2026-09-25 19:07:27 on dist-test-slave-rn98"
server_key: "21f877945f38a77a0596af18d2e170e9"
server_key_iv: "e135c24a5820a29db39ed430c63757e7"
server_key_version: "encryptionkey@0"
I20260925 19:07:28.457090 15028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:07:28.477995 15028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:07:28.480357 15028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:07:28.480536 15028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:07:28.484972 15028 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:33023
I20260925 19:07:28.485018 15096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:33023 every 8 connection(s)
I20260925 19:07:28.485462 15028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:07:28.487167 15097 sys_catalog.cc:263] Verifying existing consensus state
I20260925 19:07:28.487802 15097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc: Bootstrap starting.
I20260925 19:07:28.490756 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 15028
I20260925 19:07:28.491034 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:44299
--local_ip_for_outbound_sockets=127.0.196.193
--tserver_master_addrs=127.0.196.254:33023
--webserver_port=40425
--webserver_interface=127.0.196.193
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:39553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:07:28.496106 15097 log.cc:824] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc: Log is configured to *not* fsync() on all Append() calls
I20260925 19:07:28.498104 15097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:07:28.498432 15097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc: Bootstrap complete.
I20260925 19:07:28.500824 15097 raft_consensus.cc:348] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadd875311e24529834eae46522bebcc" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33023 } }
I20260925 19:07:28.501089 15097 raft_consensus.cc:729] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eadd875311e24529834eae46522bebcc, State: Initialized, Role: FOLLOWER
I20260925 19:07:28.501262 15097 consensus_queue.cc:260] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadd875311e24529834eae46522bebcc" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33023 } }
I20260925 19:07:28.501340 15097 raft_consensus.cc:388] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:07:28.501399 15097 raft_consensus.cc:482] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:07:28.501447 15097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [term 1 FOLLOWER]: Advancing to term 2
I20260925 19:07:28.502475 15097 raft_consensus.cc:504] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadd875311e24529834eae46522bebcc" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33023 } }
I20260925 19:07:28.502599 15097 leader_election.cc:302] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eadd875311e24529834eae46522bebcc; no voters:
I20260925 19:07:28.502825 15097 leader_election.cc:288] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 19:07:28.502864 15101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [term 2 FOLLOWER]: Leader election won for term 2
I20260925 19:07:28.502985 15101 raft_consensus.cc:686] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [term 2 LEADER]: Becoming Leader. State: Replica: eadd875311e24529834eae46522bebcc, State: Running, Role: LEADER
I20260925 19:07:28.503073 15101 consensus_queue.cc:237] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadd875311e24529834eae46522bebcc" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33023 } }
I20260925 19:07:28.503103 15097 sys_catalog.cc:564] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:07:28.503687 15103 sys_catalog.cc:455] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader eadd875311e24529834eae46522bebcc. Latest consensus state: current_term: 2 leader_uuid: "eadd875311e24529834eae46522bebcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadd875311e24529834eae46522bebcc" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33023 } } }
I20260925 19:07:28.503801 15103 sys_catalog.cc:458] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [sys.catalog]: This master's current role is: LEADER
I20260925 19:07:28.503682 15102 sys_catalog.cc:455] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "eadd875311e24529834eae46522bebcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadd875311e24529834eae46522bebcc" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33023 } } }
I20260925 19:07:28.504022 15102 sys_catalog.cc:458] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [sys.catalog]: This master's current role is: LEADER
I20260925 19:07:28.504503 15110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:07:28.505388 15110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:07:28.505492 15110 catalog_manager.cc:1259] Loaded cluster ID: 3fc3175a784f48ecb81a0d8e457e7603
I20260925 19:07:28.505523 15110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:07:28.505745 15110 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 19:07:28.510607 15110 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 19:07:28.513679 15110 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:07:28.513919 15110 catalog_manager.cc:5879] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc: Loaded TSK: 0
I20260925 19:07:28.514158 15110 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 19:07:28.576689 15099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:07:28.576947 15099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:07:28.576990 15099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:07:28.578275 15099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:07:28.578354 15099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:07:28.578401 15099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:07:28.580060 15099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:39553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:44299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=40425
--tserver_master_addrs=127.0.196.254:33023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:07:28.580358 15099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:07:28.580615 15099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:07:28.586730 15125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:28.586733 15124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:28.586982 15099 server_base.cc:1034] running on GCE node
W20260925 19:07:28.586807 15127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:28.587383 15099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:07:28.587705 15099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:07:28.588898 15099 hybrid_clock.cc:648] HybridClock initialized: now 1790363248588883 us; error 28 us; skew 500 ppm
Sep 25 19:07:28 dist-test-slave-rn98 krb5kdc[14481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:28.591928 15099 init.cc:377] Logged in from keytab as kudu/127.0.196.193@KRBTEST.COM (short username kudu)
I20260925 19:07:28.592417 15099 webserver.cc:460] Webserver started at http://127.0.196.193:40425/ using document root <none> and password file <none>
I20260925 19:07:28.592689 15099 fs_manager.cc:359] Metadata directory not provided
I20260925 19:07:28.592782 15099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:07:28.594362 15099 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:07:28.595340 15134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:28.595603 15099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:07:28.595705 15099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "7c638a81e43148dc9b88b85642501874"
format_stamp: "Formatted at 2026-09-25 19:07:27 on dist-test-slave-rn98"
server_key: "8323b699a95b6ee25b7b8b60cc883d24"
server_key_iv: "be024b255af4912b7f144a4fb92b4f64"
server_key_version: "encryptionkey@0"
I20260925 19:07:28.596110 15099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:07:28.653955 15099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:07:28.656904 15099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:07:28.657104 15099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:07:28.657413 15099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:07:28.657922 15141 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 19:07:28.659574 15099 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 19:07:28.659660 15099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 19:07:28.659711 15099 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 19:07:28.660486 15099 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 19:07:28.660552 15099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 19:07:28.660646 15141 tablet_bootstrap.cc:492] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874: Bootstrap starting.
I20260925 19:07:28.667897 15099 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:44299
I20260925 19:07:28.667979 15248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:44299 every 8 connection(s)
I20260925 19:07:28.668345 15099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
I20260925 19:07:28.672365 15141 log.cc:824] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874: Log is configured to *not* fsync() on all Append() calls
I20260925 19:07:28.674571 15141 tablet_bootstrap.cc:492] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:07:28.674947 15141 tablet_bootstrap.cc:492] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874: Bootstrap complete.
I20260925 19:07:28.676072 15141 ts_tablet_manager.cc:1397] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.010s
Sep 25 19:07:28 dist-test-slave-rn98 krb5kdc[14481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:07:28.677379 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 15099
I20260925 19:07:28.677732 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:42299
--local_ip_for_outbound_sockets=127.0.196.194
--tserver_master_addrs=127.0.196.254:33023
--webserver_port=32849
--webserver_interface=127.0.196.194
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:39553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:07:28.678380 15141 raft_consensus.cc:348] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } }
I20260925 19:07:28.678742 15141 raft_consensus.cc:729] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c638a81e43148dc9b88b85642501874, State: Initialized, Role: FOLLOWER
I20260925 19:07:28.678877 15141 consensus_queue.cc:260] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } }
I20260925 19:07:28.679230 15141 ts_tablet_manager.cc:1428] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260925 19:07:28.682418 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:28.669069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.193:35627)
0925 19:07:28.669267 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:28.669312 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:28.670102 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:28.670648 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:28.670654 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:28.671143 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:07:28.671467 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:28.671475 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.672715 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.672717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:28.673539 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:28.673542 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.674198 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.674199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:28.674342 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:28.675010 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:07:28.675020 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:07:28.676746 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:07:28.680440 (+ 3694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:28.680448 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:28.680450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:28.680793 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:28.681160 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:28.681161 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:28.681161 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:28.681282 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:28.681725 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:07:28.681731 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:07:28.681873 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 19:07:28.682004 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 19:07:28.682202 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":113,"threads_started":1}
I20260925 19:07:28.682418 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:28.669247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:35627 (local address 127.0.196.254:33023)
0925 19:07:28.669441 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:28.669450 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:28.669488 (+ 38us) server_negotiation.cc:407] Connection header received
0925 19:07:28.670335 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:28.670341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:28.670390 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:28.670509 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:28.671595 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.672565 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.673652 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.674079 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.674209 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:28.676866 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:28.676871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:28.676871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:28.676884 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:28.680323 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:28.680944 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:28.680947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:28.680948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:28.681061 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:28.681364 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:28.681365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:28.681365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:28.681638 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:28.681735 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:28.682075 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 19:07:28.682223 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":119,"threads_started":1}
I20260925 19:07:28.683357 15249 heartbeater.cc:347] Connected to a master server at 127.0.196.254:33023
I20260925 19:07:28.683439 15249 heartbeater.cc:464] Registering TS with master...
I20260925 19:07:28.683642 15249 heartbeater.cc:511] Master 127.0.196.254:33023 requested a full tablet report, sending...
I20260925 19:07:28.684446 15061 ts_manager.cc:194] Registered new tserver with Master: 7c638a81e43148dc9b88b85642501874 (127.0.196.193:44299)
I20260925 19:07:28.685799 15061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.193@KRBTEST.COM'} at 127.0.196.193:35627
I20260925 19:07:28.686741 15249 heartbeater.cc:503] Master 127.0.196.254:33023 was elected leader, sending a full tablet report...
I20260925 19:07:28.689774 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:28.685161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:44299 (local address 127.0.0.1:42360)
0925 19:07:28.685348 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:28.685391 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:28.685446 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:28.686958 (+ 1512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:28.686961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:28.686964 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:07:28.687039 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:28.687041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.687940 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.687941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:28.688755 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:28.688757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.689411 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.689412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:28.689508 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:28.689521 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 19:07:28.689620 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 19:07:28.689686 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":125,"threads_started":1}
I20260925 19:07:28.689780 15257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:28.685214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42360 (local address 127.0.196.193:44299)
0925 19:07:28.686749 (+ 1535us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:28.686754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:28.686772 (+ 18us) server_negotiation.cc:407] Connection header received
0925 19:07:28.686811 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:28.686814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:28.686832 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:28.686926 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:07:28.687121 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.687852 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.688857 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.689323 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.689410 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:28.689470 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:28.689650 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 19:07:28.689689 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1496,"thread_start_us":88,"threads_started":1}
I20260925 19:07:28.690235 15183 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bf8d0e6bca74c49a1f92891a16cd6df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:07:28 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:42360
I20260925 19:07:28.690923 15258 tablet_replica.cc:330] stopping tablet replica
I20260925 19:07:28.691054 15258 raft_consensus.cc:2229] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:07:28.691141 15258 raft_consensus.cc:2258] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:07:28.691478 15258 ts_tablet_manager.cc:1905] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:07:28.693750 15258 ts_tablet_manager.cc:1918] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:07:28.693868 15258 log.cc:1196] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/wal/wals/9bf8d0e6bca74c49a1f92891a16cd6df
I20260925 19:07:28.694119 15258 ts_tablet_manager.cc:1939] T 9bf8d0e6bca74c49a1f92891a16cd6df P 7c638a81e43148dc9b88b85642501874: Deleting consensus metadata
I20260925 19:07:28.694746 15048 catalog_manager.cc:4841] TS 7c638a81e43148dc9b88b85642501874 (127.0.196.193:44299): tablet 9bf8d0e6bca74c49a1f92891a16cd6df (table test-table [id=9f0da79eda654244b1a72cacb86bbbe3]) successfully deleted
W20260925 19:07:28.764497 15254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:07:28.764710 15254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:07:28.764735 15254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:07:28.765928 15254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:07:28.765973 15254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:07:28.766004 15254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:07:28.767578 15254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:39553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:42299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=32849
--tserver_master_addrs=127.0.196.254:33023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:07:28.767830 15254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:07:28.768091 15254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:07:28.773229 15264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:28.773254 15266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:28.773363 15254 server_base.cc:1034] running on GCE node
W20260925 19:07:28.773437 15263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:28.773756 15254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:07:28.773999 15254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:07:28.775146 15254 hybrid_clock.cc:648] HybridClock initialized: now 1790363248775126 us; error 41 us; skew 500 ppm
Sep 25 19:07:28 dist-test-slave-rn98 krb5kdc[14481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:28.778079 15254 init.cc:377] Logged in from keytab as kudu/127.0.196.194@KRBTEST.COM (short username kudu)
I20260925 19:07:28.778723 15254 webserver.cc:460] Webserver started at http://127.0.196.194:32849/ using document root <none> and password file <none>
I20260925 19:07:28.778959 15254 fs_manager.cc:359] Metadata directory not provided
I20260925 19:07:28.779026 15254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:07:28.780606 15254 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 19:07:28.781296 15273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:28.781599 15254 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:07:28.781697 15254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "1eb37032863941f18f46907b9d7a07dc"
format_stamp: "Formatted at 2026-09-25 19:07:27 on dist-test-slave-rn98"
server_key: "48057dd7b0d0d70f9689ef5c7c3b2398"
server_key_iv: "81c872f7cffceb5bcc9a6dc2fa8e1f2d"
server_key_version: "encryptionkey@0"
I20260925 19:07:28.782069 15254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:07:28.792518 15254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:07:28.795217 15254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:07:28.795392 15254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:07:28.795670 15254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:07:28.796016 15254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:07:28.796078 15254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:28.796123 15254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:07:28.796155 15254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:28.802865 15254 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:42299
I20260925 19:07:28.802935 15386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:42299 every 8 connection(s)
I20260925 19:07:28.803442 15254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:07:28 dist-test-slave-rn98 krb5kdc[14481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:07:28.813380 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 15254
I20260925 19:07:28.813724 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:44467
--local_ip_for_outbound_sockets=127.0.196.195
--tserver_master_addrs=127.0.196.254:33023
--webserver_port=41353
--webserver_interface=127.0.196.195
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:39553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:07:28.815171 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:28.804413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:47863 (local address 127.0.196.254:33023)
0925 19:07:28.804494 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:28.804496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:28.804826 (+ 330us) server_negotiation.cc:407] Connection header received
0925 19:07:28.805457 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:28.805458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:28.805479 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:28.805534 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:28.806605 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.807212 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.808354 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.808616 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.808714 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:28.811471 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:28.811473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:28.811473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:28.811481 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:28.813342 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:28.814007 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:28.814009 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:28.814009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:28.814078 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:28.814376 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:28.814377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:28.814378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:28.814521 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:28.814602 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:28.814970 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 19:07:28.815079 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 19:07:28.815383 15389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:28.804401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.194:47863)
0925 19:07:28.804678 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:28.804717 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:28.805333 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:28.805630 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:28.805634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:28.806253 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:07:28.806484 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:28.806489 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.807332 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.807334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:28.808244 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:28.808247 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.808728 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.808729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:28.808881 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:28.809586 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:07:28.809596 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:07:28.811361 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:07:28.813552 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:28.813568 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:28.813570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:28.813886 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:28.814152 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:28.814153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:28.814154 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:28.814283 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:28.814606 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:07:28.814614 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:07:28.814826 (+ 212us) client_negotiation.cc:769] Sending connection context
0925 19:07:28.814944 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 19:07:28.815121 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":182,"threads_started":1}
I20260925 19:07:28.815910 15387 heartbeater.cc:347] Connected to a master server at 127.0.196.254:33023
I20260925 19:07:28.816045 15387 heartbeater.cc:464] Registering TS with master...
I20260925 19:07:28.816293 15387 heartbeater.cc:511] Master 127.0.196.254:33023 requested a full tablet report, sending...
I20260925 19:07:28.816823 15061 ts_manager.cc:194] Registered new tserver with Master: 1eb37032863941f18f46907b9d7a07dc (127.0.196.194:42299)
I20260925 19:07:28.817245 15061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.194@KRBTEST.COM'} at 127.0.196.194:47863
W20260925 19:07:28.907395 15390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:07:28.907671 15390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:07:28.907718 15390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:07:28.909049 15390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:07:28.909126 15390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:07:28.909179 15390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:07:28.910943 15390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:39553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:44467
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=41353
--tserver_master_addrs=127.0.196.254:33023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:07:28.911247 15390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:07:28.911504 15390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:07:28.917129 15395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:28.917167 15398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:28.917135 15396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:28.917532 15390 server_base.cc:1034] running on GCE node
I20260925 19:07:28.917757 15390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:07:28.918026 15390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:07:28.919210 15390 hybrid_clock.cc:648] HybridClock initialized: now 1790363248919188 us; error 40 us; skew 500 ppm
Sep 25 19:07:28 dist-test-slave-rn98 krb5kdc[14481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:28.921981 15390 init.cc:377] Logged in from keytab as kudu/127.0.196.195@KRBTEST.COM (short username kudu)
I20260925 19:07:28.922550 15390 webserver.cc:460] Webserver started at http://127.0.196.195:41353/ using document root <none> and password file <none>
I20260925 19:07:28.922858 15390 fs_manager.cc:359] Metadata directory not provided
I20260925 19:07:28.922958 15390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:07:28.924544 15390 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:07:28.925177 15405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:28.925369 15390 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:07:28.925549 15390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "04f233d4c02b413daf423a661674cd48"
format_stamp: "Formatted at 2026-09-25 19:07:27 on dist-test-slave-rn98"
server_key: "8ac3f86a1be2cd8a1a4d3cc5c89d3de8"
server_key_iv: "2e1b15cb0a9a9a80b0bd66d5e9420906"
server_key_version: "encryptionkey@0"
I20260925 19:07:28.925969 15390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:07:28.943161 15390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:07:28.945175 15390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:07:28.945354 15390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:07:28.945598 15390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:07:28.945916 15390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:07:28.945972 15390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:28.946005 15390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:07:28.946035 15390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:28.952067 15390 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:44467
I20260925 19:07:28.952119 15518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:44467 every 8 connection(s)
I20260925 19:07:28.952479 15390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
Sep 25 19:07:28 dist-test-slave-rn98 krb5kdc[14481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:07:28.960165 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 15390
I20260925 19:07:28.962888 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:28.953289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:45075 (local address 127.0.196.254:33023)
0925 19:07:28.953367 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:28.953369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:28.953607 (+ 238us) server_negotiation.cc:407] Connection header received
0925 19:07:28.954137 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:28.954138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:28.954149 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:28.954203 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:28.955322 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.955823 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.956882 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.957084 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.957162 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:28.959678 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:28.959680 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:28.959680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:28.959689 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:28.961368 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:28.961913 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:28.961914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:28.961914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:28.961978 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:28.962222 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:28.962223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:28.962223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:28.962345 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:28.962409 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:28.962754 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 19:07:28.962823 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 19:07:28.963163 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:28.953296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.195:45075)
0925 19:07:28.953466 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:28.953503 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:28.954035 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:28.954285 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:28.954289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:28.955005 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:07:28.955213 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:28.955218 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.955917 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.955918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:28.956795 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:28.956797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.957176 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.957176 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:28.957278 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:28.957947 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:07:28.957956 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:07:28.959454 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:07:28.961455 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:28.961465 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:28.961469 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:28.961843 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:28.962049 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:28.962049 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:28.962049 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:28.962159 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:28.962406 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:07:28.962411 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:07:28.962564 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 19:07:28.962729 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 19:07:28.962910 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260925 19:07:28.963653 15519 heartbeater.cc:347] Connected to a master server at 127.0.196.254:33023
I20260925 19:07:28.963757 15519 heartbeater.cc:464] Registering TS with master...
I20260925 19:07:28.963995 15519 heartbeater.cc:511] Master 127.0.196.254:33023 requested a full tablet report, sending...
I20260925 19:07:28.964430 15061 ts_manager.cc:194] Registered new tserver with Master: 04f233d4c02b413daf423a661674cd48 (127.0.196.195:44467)
I20260925 19:07:28.964764 15061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.195@KRBTEST.COM'} at 127.0.196.195:45075
I20260925 19:07:28.968401 15522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:28.960588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45662 (local address 127.0.196.254:33023)
0925 19:07:28.960753 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:28.960755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:28.960846 (+ 91us) server_negotiation.cc:407] Connection header received
0925 19:07:28.960928 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:28.960929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:28.960941 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:28.961032 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:28.961738 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.962490 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.963608 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.963871 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.963930 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:28.965836 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:28.965837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:28.965838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:28.965844 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:28.967274 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:28.967667 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:28.967668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:28.967669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:28.967716 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:28.967951 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:28.967952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:28.967952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:28.968047 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:28.968187 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:28.968270 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 19:07:28.968335 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":66,"threads_started":1}
I20260925 19:07:28.969236 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:07:28.972157 15522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:28.969666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45678 (local address 127.0.196.254:33023)
0925 19:07:28.969719 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:28.969721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:28.969845 (+ 124us) server_negotiation.cc:407] Connection header received
0925 19:07:28.969894 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:28.969894 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:28.969921 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:28.969979 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:07:28.970063 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.970527 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.971411 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.971601 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.971683 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:28.971872 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:07:28.971968 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:07:28.972034 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:28.972081 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 19:07:28.972101 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 19:07:28.972415 15061 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:07:28.973994 15061 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:07:28.976266 15183 tablet_service.cc:1467] Processing CreateTablet for tablet c8e8fe384a3441e0b9285a8d86d142cf (DEFAULT_TABLE table=test-table [id=6d48f258b7414f91b185b64862ba912d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:07:28.976482 15183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8e8fe384a3441e0b9285a8d86d142cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:07:28.978025 15141 tablet_bootstrap.cc:492] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874: Bootstrap starting.
I20260925 19:07:28.978356 15141 tablet_bootstrap.cc:654] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874: Neither blocks nor log segments found. Creating new log.
I20260925 19:07:28.979058 15141 tablet_bootstrap.cc:492] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874: No bootstrap required, opened a new log
I20260925 19:07:28.979104 15141 ts_tablet_manager.cc:1397] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 19:07:28.979230 15141 raft_consensus.cc:348] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } }
I20260925 19:07:28.979300 15141 raft_consensus.cc:374] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:07:28.979367 15141 raft_consensus.cc:729] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c638a81e43148dc9b88b85642501874, State: Initialized, Role: FOLLOWER
I20260925 19:07:28.979424 15141 consensus_queue.cc:260] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } }
I20260925 19:07:28.979521 15141 ts_tablet_manager.cc:1428] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:28.980513 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:28.975942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:42299 (local address 127.0.0.1:43428)
0925 19:07:28.975984 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:28.975996 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:28.976035 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:28.976451 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:28.976452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:28.976528 (+ 76us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:07:28.976573 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:28.976575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.977833 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.977833 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:28.978852 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:28.978854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.980267 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.980268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:28.980322 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:28.980347 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 19:07:28.980399 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 19:07:28.980450 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 19:07:28.980773 15526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:28.976071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43428 (local address 127.0.196.194:42299)
0925 19:07:28.976249 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:28.976255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:28.976266 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:07:28.976314 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:28.976316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:28.976328 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:28.976454 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:07:28.976642 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.977714 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.978940 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.980092 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.980161 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:28.980225 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:28.980652 (+ 427us) server_negotiation.cc:299] Negotiation successful
0925 19:07:28.980693 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":100,"threads_started":1}
I20260925 19:07:28.981400 15321 tablet_service.cc:1467] Processing CreateTablet for tablet c8e8fe384a3441e0b9285a8d86d142cf (DEFAULT_TABLE table=test-table [id=6d48f258b7414f91b185b64862ba912d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:07:28.981777 15321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8e8fe384a3441e0b9285a8d86d142cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:07:28.982033 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:28.976045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:44467 (local address 127.0.0.1:50404)
0925 19:07:28.976158 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:28.976167 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:28.976221 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:28.978507 (+ 2286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:28.978508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:28.978511 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:07:28.978552 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:28.978554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.980017 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.980017 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:28.981028 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:28.981030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.981777 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.981778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:28.981825 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:28.981828 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:07:28.981900 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 19:07:28.981943 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":76,"threads_started":1}
I20260925 19:07:28.982227 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:28.978101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50404 (local address 127.0.196.195:44467)
0925 19:07:28.978313 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:28.978318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:28.978330 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:07:28.978373 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:28.978375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:28.978392 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:28.978496 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:07:28.978733 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.979843 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.981110 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:28.981594 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:28.981683 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:28.981745 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:28.981925 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 19:07:28.981975 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":97,"threads_started":1}
I20260925 19:07:28.982999 15453 tablet_service.cc:1467] Processing CreateTablet for tablet c8e8fe384a3441e0b9285a8d86d142cf (DEFAULT_TABLE table=test-table [id=6d48f258b7414f91b185b64862ba912d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:07:28.983335 15453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8e8fe384a3441e0b9285a8d86d142cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:07:28.985569 15529 tablet_bootstrap.cc:492] T c8e8fe384a3441e0b9285a8d86d142cf P 1eb37032863941f18f46907b9d7a07dc: Bootstrap starting.
I20260925 19:07:28.986266 15529 tablet_bootstrap.cc:654] T c8e8fe384a3441e0b9285a8d86d142cf P 1eb37032863941f18f46907b9d7a07dc: Neither blocks nor log segments found. Creating new log.
I20260925 19:07:28.986883 15529 log.cc:824] T c8e8fe384a3441e0b9285a8d86d142cf P 1eb37032863941f18f46907b9d7a07dc: Log is configured to *not* fsync() on all Append() calls
I20260925 19:07:28.987124 15530 tablet_bootstrap.cc:492] T c8e8fe384a3441e0b9285a8d86d142cf P 04f233d4c02b413daf423a661674cd48: Bootstrap starting.
I20260925 19:07:28.987711 15530 tablet_bootstrap.cc:654] T c8e8fe384a3441e0b9285a8d86d142cf P 04f233d4c02b413daf423a661674cd48: Neither blocks nor log segments found. Creating new log.
I20260925 19:07:28.987779 15529 tablet_bootstrap.cc:492] T c8e8fe384a3441e0b9285a8d86d142cf P 1eb37032863941f18f46907b9d7a07dc: No bootstrap required, opened a new log
I20260925 19:07:28.987851 15529 ts_tablet_manager.cc:1397] T c8e8fe384a3441e0b9285a8d86d142cf P 1eb37032863941f18f46907b9d7a07dc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 19:07:28.988178 15530 log.cc:824] T c8e8fe384a3441e0b9285a8d86d142cf P 04f233d4c02b413daf423a661674cd48: Log is configured to *not* fsync() on all Append() calls
I20260925 19:07:28.989151 15530 tablet_bootstrap.cc:492] T c8e8fe384a3441e0b9285a8d86d142cf P 04f233d4c02b413daf423a661674cd48: No bootstrap required, opened a new log
I20260925 19:07:28.989208 15530 ts_tablet_manager.cc:1397] T c8e8fe384a3441e0b9285a8d86d142cf P 04f233d4c02b413daf423a661674cd48: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:07:28.989508 15529 raft_consensus.cc:348] T c8e8fe384a3441e0b9285a8d86d142cf P 1eb37032863941f18f46907b9d7a07dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } }
I20260925 19:07:28.989675 15529 raft_consensus.cc:374] T c8e8fe384a3441e0b9285a8d86d142cf P 1eb37032863941f18f46907b9d7a07dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:07:28.989723 15529 raft_consensus.cc:729] T c8e8fe384a3441e0b9285a8d86d142cf P 1eb37032863941f18f46907b9d7a07dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb37032863941f18f46907b9d7a07dc, State: Initialized, Role: FOLLOWER
I20260925 19:07:28.989869 15529 consensus_queue.cc:260] T c8e8fe384a3441e0b9285a8d86d142cf P 1eb37032863941f18f46907b9d7a07dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } }
I20260925 19:07:28.990157 15529 ts_tablet_manager.cc:1428] T c8e8fe384a3441e0b9285a8d86d142cf P 1eb37032863941f18f46907b9d7a07dc: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 19:07:28.990222 15387 heartbeater.cc:503] Master 127.0.196.254:33023 was elected leader, sending a full tablet report...
I20260925 19:07:28.990986 15530 raft_consensus.cc:348] T c8e8fe384a3441e0b9285a8d86d142cf P 04f233d4c02b413daf423a661674cd48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } }
I20260925 19:07:28.991168 15530 raft_consensus.cc:374] T c8e8fe384a3441e0b9285a8d86d142cf P 04f233d4c02b413daf423a661674cd48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:07:28.991216 15530 raft_consensus.cc:729] T c8e8fe384a3441e0b9285a8d86d142cf P 04f233d4c02b413daf423a661674cd48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f233d4c02b413daf423a661674cd48, State: Initialized, Role: FOLLOWER
I20260925 19:07:28.991338 15530 consensus_queue.cc:260] T c8e8fe384a3441e0b9285a8d86d142cf P 04f233d4c02b413daf423a661674cd48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } }
I20260925 19:07:28.991647 15530 ts_tablet_manager.cc:1428] T c8e8fe384a3441e0b9285a8d86d142cf P 04f233d4c02b413daf423a661674cd48: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 19:07:28.991662 15519 heartbeater.cc:503] Master 127.0.196.254:33023 was elected leader, sending a full tablet report...
W20260925 19:07:29.054105 15388 tablet.cc:2369] T c8e8fe384a3441e0b9285a8d86d142cf P 1eb37032863941f18f46907b9d7a07dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:07:29.169198 15250 tablet.cc:2369] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:07:29.203172 15520 tablet.cc:2369] T c8e8fe384a3441e0b9285a8d86d142cf P 04f233d4c02b413daf423a661674cd48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:07:29.257517 15255 raft_consensus.cc:482] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:07:29.257674 15255 raft_consensus.cc:504] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } }
I20260925 19:07:29.258076 15255 leader_election.cc:288] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1eb37032863941f18f46907b9d7a07dc (127.0.196.194:42299), 04f233d4c02b413daf423a661674cd48 (127.0.196.195:44467)
I20260925 19:07:29.261945 15526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:29.258464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:43725 (local address 127.0.196.194:42299)
0925 19:07:29.258518 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:29.258520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:29.258636 (+ 116us) server_negotiation.cc:407] Connection header received
0925 19:07:29.258731 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:29.258732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:29.258744 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:29.258812 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:07:29.259068 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:29.259674 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:29.261166 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:29.261543 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:29.261649 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:29.261660 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:29.261812 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 19:07:29.261857 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 19:07:29.261986 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:29.258745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:55577 (local address 127.0.196.195:44467)
0925 19:07:29.258832 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:29.258835 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:29.258849 (+ 14us) server_negotiation.cc:407] Connection header received
0925 19:07:29.258878 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:29.258879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:29.258894 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:29.258976 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:07:29.259154 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:29.259798 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:29.261069 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:29.261546 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:29.261651 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:29.261660 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:29.261856 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 19:07:29.261903 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 19:07:29.261992 15535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:29.258385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:42299 (local address 127.0.196.193:43725)
0925 19:07:29.258528 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:29.258542 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:29.258640 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:29.258904 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:29.258905 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:29.258908 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:07:29.258976 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:29.258978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:29.259798 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:29.259799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:29.261070 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:29.261073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:29.261644 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:29.261645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:29.261718 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:29.261723 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:07:29.261784 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 19:07:29.261859 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":30,"thread_start_us":75,"threads_started":1}
I20260925 19:07:29.261999 15536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:29.258605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:44467 (local address 127.0.196.193:55577)
0925 19:07:29.258732 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:29.258743 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:29.258817 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:29.258982 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:29.258983 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:29.258985 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:07:29.259066 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:29.259068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:29.259923 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:29.259924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:29.260973 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:29.260976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:29.261670 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:29.261671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:29.261735 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:29.261739 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:07:29.261829 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 19:07:29.261869 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
I20260925 19:07:29.262727 15341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e8fe384a3441e0b9285a8d86d142cf" candidate_uuid: "7c638a81e43148dc9b88b85642501874" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb37032863941f18f46907b9d7a07dc" is_pre_election: true
I20260925 19:07:29.262727 15473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e8fe384a3441e0b9285a8d86d142cf" candidate_uuid: "7c638a81e43148dc9b88b85642501874" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f233d4c02b413daf423a661674cd48" is_pre_election: true
I20260925 19:07:29.262916 15341 raft_consensus.cc:2454] T c8e8fe384a3441e0b9285a8d86d142cf P 1eb37032863941f18f46907b9d7a07dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c638a81e43148dc9b88b85642501874 in term 0.
I20260925 19:07:29.262923 15473 raft_consensus.cc:2454] T c8e8fe384a3441e0b9285a8d86d142cf P 04f233d4c02b413daf423a661674cd48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c638a81e43148dc9b88b85642501874 in term 0.
I20260925 19:07:29.263170 15135 leader_election.cc:302] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb37032863941f18f46907b9d7a07dc, 7c638a81e43148dc9b88b85642501874; no voters:
I20260925 19:07:29.263310 15255 raft_consensus.cc:2781] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:07:29.263403 15255 raft_consensus.cc:482] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:07:29.263443 15255 raft_consensus.cc:3037] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:07:29.264184 15255 raft_consensus.cc:504] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } }
I20260925 19:07:29.264338 15255 leader_election.cc:288] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [CANDIDATE]: Term 1 election: Requested vote from peers 1eb37032863941f18f46907b9d7a07dc (127.0.196.194:42299), 04f233d4c02b413daf423a661674cd48 (127.0.196.195:44467)
I20260925 19:07:29.264564 15341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e8fe384a3441e0b9285a8d86d142cf" candidate_uuid: "7c638a81e43148dc9b88b85642501874" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb37032863941f18f46907b9d7a07dc"
I20260925 19:07:29.264607 15473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e8fe384a3441e0b9285a8d86d142cf" candidate_uuid: "7c638a81e43148dc9b88b85642501874" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f233d4c02b413daf423a661674cd48"
I20260925 19:07:29.264660 15341 raft_consensus.cc:3037] T c8e8fe384a3441e0b9285a8d86d142cf P 1eb37032863941f18f46907b9d7a07dc [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:07:29.264678 15473 raft_consensus.cc:3037] T c8e8fe384a3441e0b9285a8d86d142cf P 04f233d4c02b413daf423a661674cd48 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:07:29.265383 15341 raft_consensus.cc:2454] T c8e8fe384a3441e0b9285a8d86d142cf P 1eb37032863941f18f46907b9d7a07dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c638a81e43148dc9b88b85642501874 in term 1.
I20260925 19:07:29.265533 15473 raft_consensus.cc:2454] T c8e8fe384a3441e0b9285a8d86d142cf P 04f233d4c02b413daf423a661674cd48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c638a81e43148dc9b88b85642501874 in term 1.
I20260925 19:07:29.265753 15135 leader_election.cc:302] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04f233d4c02b413daf423a661674cd48, 7c638a81e43148dc9b88b85642501874; no voters:
I20260925 19:07:29.265892 15255 raft_consensus.cc:2781] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:07:29.266081 15255 raft_consensus.cc:686] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [term 1 LEADER]: Becoming Leader. State: Replica: 7c638a81e43148dc9b88b85642501874, State: Running, Role: LEADER
I20260925 19:07:29.266145 15255 consensus_queue.cc:237] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } }
I20260925 19:07:29.266954 15060 catalog_manager.cc:5495] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c638a81e43148dc9b88b85642501874 (127.0.196.193). New cstate: current_term: 1 leader_uuid: "7c638a81e43148dc9b88b85642501874" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } health_report { overall_health: HEALTHY } } }
I20260925 19:07:29.335773 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:29.332783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42372 (local address 127.0.196.193:44299)
0925 19:07:29.332939 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:29.332941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:29.332947 (+ 6us) server_negotiation.cc:407] Connection header received
0925 19:07:29.332986 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:29.332987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:29.333007 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:29.333142 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:07:29.333284 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:29.333805 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:29.334639 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:29.334832 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:29.334909 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:29.335378 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:07:29.335489 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:07:29.335592 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:29.335637 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 19:07:29.335662 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":74,"threads_started":1}
I20260925 19:07:29.337423 15341 raft_consensus.cc:1261] T c8e8fe384a3441e0b9285a8d86d142cf P 1eb37032863941f18f46907b9d7a07dc [term 1 FOLLOWER]: Refusing update from remote peer 7c638a81e43148dc9b88b85642501874: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:07:29.337455 15473 raft_consensus.cc:1261] T c8e8fe384a3441e0b9285a8d86d142cf P 04f233d4c02b413daf423a661674cd48 [term 1 FOLLOWER]: Refusing update from remote peer 7c638a81e43148dc9b88b85642501874: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:07:29.337841 15255 consensus_queue.cc:1035] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:07:29.338033 15537 consensus_queue.cc:1035] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:07:29.340631 15542 mvcc.cc:205] Tried to move back new op lower bound from 7333327869283389440 to 7333327868994777088. Current Snapshot: MvccSnapshot[applied={T|T < 7333327869283389440}]
I20260925 19:07:29.340798 15543 mvcc.cc:205] Tried to move back new op lower bound from 7333327869283389440 to 7333327868994777088. Current Snapshot: MvccSnapshot[applied={T|T < 7333327869283389440}]
I20260925 19:07:29.342852 15061 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:07:29.342958 15061 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:07:29.343866 15061 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc: Sending DeleteTablet for 3 replicas of tablet c8e8fe384a3441e0b9285a8d86d142cf
I20260925 19:07:29.344264 15183 tablet_service.cc:1514] Processing DeleteTablet for tablet c8e8fe384a3441e0b9285a8d86d142cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:07:29 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:42360
I20260925 19:07:29.344269 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 15099
I20260925 19:07:29.344259 15321 tablet_service.cc:1514] Processing DeleteTablet for tablet c8e8fe384a3441e0b9285a8d86d142cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:07:29 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:43428
I20260925 19:07:29.344472 15453 tablet_service.cc:1514] Processing DeleteTablet for tablet c8e8fe384a3441e0b9285a8d86d142cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:07:29 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:50404
I20260925 19:07:29.345064 15550 tablet_replica.cc:330] T c8e8fe384a3441e0b9285a8d86d142cf P 1eb37032863941f18f46907b9d7a07dc: stopping tablet replica
I20260925 19:07:29.345183 15550 raft_consensus.cc:2229] T c8e8fe384a3441e0b9285a8d86d142cf P 1eb37032863941f18f46907b9d7a07dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:07:29.345322 15550 raft_consensus.cc:2258] T c8e8fe384a3441e0b9285a8d86d142cf P 1eb37032863941f18f46907b9d7a07dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:07:29.345511 15551 tablet_replica.cc:330] T c8e8fe384a3441e0b9285a8d86d142cf P 04f233d4c02b413daf423a661674cd48: stopping tablet replica
I20260925 19:07:29.345613 15551 raft_consensus.cc:2229] T c8e8fe384a3441e0b9285a8d86d142cf P 04f233d4c02b413daf423a661674cd48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:07:29.345713 15551 raft_consensus.cc:2258] T c8e8fe384a3441e0b9285a8d86d142cf P 04f233d4c02b413daf423a661674cd48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:07:29.345829 15550 ts_tablet_manager.cc:1905] T c8e8fe384a3441e0b9285a8d86d142cf P 1eb37032863941f18f46907b9d7a07dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:07:29.346140 15551 ts_tablet_manager.cc:1905] T c8e8fe384a3441e0b9285a8d86d142cf P 04f233d4c02b413daf423a661674cd48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:07:29.347560 15550 ts_tablet_manager.cc:1918] T c8e8fe384a3441e0b9285a8d86d142cf P 1eb37032863941f18f46907b9d7a07dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:07:29.347644 15550 log.cc:1196] T c8e8fe384a3441e0b9285a8d86d142cf P 1eb37032863941f18f46907b9d7a07dc: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/wal/wals/c8e8fe384a3441e0b9285a8d86d142cf
I20260925 19:07:29.347831 15551 ts_tablet_manager.cc:1918] T c8e8fe384a3441e0b9285a8d86d142cf P 04f233d4c02b413daf423a661674cd48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:07:29.347901 15551 log.cc:1196] T c8e8fe384a3441e0b9285a8d86d142cf P 04f233d4c02b413daf423a661674cd48: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/wal/wals/c8e8fe384a3441e0b9285a8d86d142cf
I20260925 19:07:29.347914 15550 ts_tablet_manager.cc:1939] T c8e8fe384a3441e0b9285a8d86d142cf P 1eb37032863941f18f46907b9d7a07dc: Deleting consensus metadata
I20260925 19:07:29.348202 15551 ts_tablet_manager.cc:1939] T c8e8fe384a3441e0b9285a8d86d142cf P 04f233d4c02b413daf423a661674cd48: Deleting consensus metadata
I20260925 19:07:29.348434 15045 catalog_manager.cc:4841] TS 1eb37032863941f18f46907b9d7a07dc (127.0.196.194:42299): tablet c8e8fe384a3441e0b9285a8d86d142cf (table test-table [id=6d48f258b7414f91b185b64862ba912d]) successfully deleted
I20260925 19:07:29.348603 15045 catalog_manager.cc:4841] TS 04f233d4c02b413daf423a661674cd48 (127.0.196.195:44467): tablet c8e8fe384a3441e0b9285a8d86d142cf (table test-table [id=6d48f258b7414f91b185b64862ba912d]) successfully deleted
W20260925 19:07:29.350495 15048 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.196.193:44299 (error 108)
I20260925 19:07:29.351055 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:29.350919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:44299 (local address 127.0.0.1:42388)
0925 19:07:29.350976 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:29.351020 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.193:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 19:07:29.351148 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 15254
W20260925 19:07:29.351169 15048 catalog_manager.cc:4584] TS 7c638a81e43148dc9b88b85642501874 (127.0.196.193:44299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c8e8fe384a3441e0b9285a8d86d142cf: Network error: Client connection negotiation failed: client connection to 127.0.196.193:44299: connect: Connection refused (error 111)
I20260925 19:07:29.356793 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 15390
I20260925 19:07:29.361426 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 15028
I20260925 19:07:29.366974 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:33023
--webserver_interface=127.0.196.254
--webserver_port=37197
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:39553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:33023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:07:29.457639 15553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:07:29.457893 15553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:07:29.457921 15553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:07:29.459630 15553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:07:29.459693 15553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:07:29.459709 15553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:07:29.459722 15553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:07:29.459735 15553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:07:29.461397 15553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:39553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:33023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:33023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=37197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:07:29.461695 15553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:07:29.461941 15553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:07:29.467450 15558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:29.467658 15553 server_base.cc:1034] running on GCE node
W20260925 19:07:29.467550 15561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:29.467537 15559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:29.468034 15553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:07:29.468393 15553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:07:29.469597 15553 hybrid_clock.cc:648] HybridClock initialized: now 1790363249469571 us; error 42 us; skew 500 ppm
Sep 25 19:07:29 dist-test-slave-rn98 krb5kdc[14481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:29.472410 15553 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:07:29.472832 15553 webserver.cc:460] Webserver started at http://127.0.196.254:37197/ using document root <none> and password file <none>
I20260925 19:07:29.473104 15553 fs_manager.cc:359] Metadata directory not provided
I20260925 19:07:29.473153 15553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:07:29.474668 15553 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:07:29.475431 15568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:29.475668 15553 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:07:29.475739 15553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "eadd875311e24529834eae46522bebcc"
format_stamp: "Formatted at 2026-09-25 19:07:27 on dist-test-slave-rn98"
server_key: "21f877945f38a77a0596af18d2e170e9"
server_key_iv: "e135c24a5820a29db39ed430c63757e7"
server_key_version: "encryptionkey@0"
I20260925 19:07:29.476136 15553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:07:29.497589 15553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:07:29.500205 15553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:07:29.500388 15553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:07:29.504582 15553 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:33023
I20260925 19:07:29.504631 15620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:33023 every 8 connection(s)
I20260925 19:07:29.505040 15553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:07:29.506954 15621 sys_catalog.cc:263] Verifying existing consensus state
I20260925 19:07:29.507665 15621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc: Bootstrap starting.
I20260925 19:07:29.512555 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 15553
I20260925 19:07:29.512842 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:44299
--local_ip_for_outbound_sockets=127.0.196.193
--tserver_master_addrs=127.0.196.254:33023
--webserver_port=40425
--webserver_interface=127.0.196.193
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:39553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:07:29.516414 15621 log.cc:824] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc: Log is configured to *not* fsync() on all Append() calls
I20260925 19:07:29.518959 15621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:07:29.519308 15621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc: Bootstrap complete.
I20260925 19:07:29.521641 15621 raft_consensus.cc:348] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadd875311e24529834eae46522bebcc" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33023 } }
I20260925 19:07:29.521908 15621 raft_consensus.cc:729] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: eadd875311e24529834eae46522bebcc, State: Initialized, Role: FOLLOWER
I20260925 19:07:29.522140 15621 consensus_queue.cc:260] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadd875311e24529834eae46522bebcc" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33023 } }
I20260925 19:07:29.522244 15621 raft_consensus.cc:388] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:07:29.522300 15621 raft_consensus.cc:482] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:07:29.522351 15621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [term 2 FOLLOWER]: Advancing to term 3
I20260925 19:07:29.523530 15621 raft_consensus.cc:504] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadd875311e24529834eae46522bebcc" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33023 } }
I20260925 19:07:29.523664 15621 leader_election.cc:302] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eadd875311e24529834eae46522bebcc; no voters:
I20260925 19:07:29.523872 15621 leader_election.cc:288] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 19:07:29.523914 15625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [term 3 FOLLOWER]: Leader election won for term 3
I20260925 19:07:29.524039 15625 raft_consensus.cc:686] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [term 3 LEADER]: Becoming Leader. State: Replica: eadd875311e24529834eae46522bebcc, State: Running, Role: LEADER
I20260925 19:07:29.524132 15625 consensus_queue.cc:237] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadd875311e24529834eae46522bebcc" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33023 } }
I20260925 19:07:29.524180 15621 sys_catalog.cc:564] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:07:29.524394 15625 sys_catalog.cc:455] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader eadd875311e24529834eae46522bebcc. Latest consensus state: current_term: 3 leader_uuid: "eadd875311e24529834eae46522bebcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadd875311e24529834eae46522bebcc" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33023 } } }
I20260925 19:07:29.524469 15625 sys_catalog.cc:458] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [sys.catalog]: This master's current role is: LEADER
I20260925 19:07:29.524401 15626 sys_catalog.cc:455] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "eadd875311e24529834eae46522bebcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadd875311e24529834eae46522bebcc" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33023 } } }
I20260925 19:07:29.524528 15626 sys_catalog.cc:458] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc [sys.catalog]: This master's current role is: LEADER
I20260925 19:07:29.526024 15641 catalog_manager.cc:1259] Loaded cluster ID: 3fc3175a784f48ecb81a0d8e457e7603
I20260925 19:07:29.526100 15641 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc: loading cluster ID for follower catalog manager: success
W20260925 19:07:29.526345 15641 catalog_manager.cc:1576] T 00000000000000000000000000000000 P eadd875311e24529834eae46522bebcc: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260925 19:07:29.526391 15641 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260925 19:07:29.526445 15629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:07:29.526876 15629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:07:29.526986 15629 catalog_manager.cc:1259] Loaded cluster ID: 3fc3175a784f48ecb81a0d8e457e7603
I20260925 19:07:29.527042 15629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 19:07:29.527187 15629 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9a489539ad google::LogMessage::Fail() at ??:0
@ 0x7f9a4895688a google::LogMessage::SendToLog() at ??:0
@ 0x7f9a489534de google::LogMessage::Flush() at ??:0
@ 0x7f9a48954a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9a508740f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f9a5087d8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9a4a8210d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9a4a816954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9a4b67a6db start_thread at ??:0
@ 0x7f9a4745161f clone at ??:0
W20260925 19:07:29.614872 15623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:07:29.615186 15623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:07:29.615228 15623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:07:29.617156 15623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:07:29.617224 15623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:07:29.617269 15623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:07:29.619716 15623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:39553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:44299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=40425
--tserver_master_addrs=127.0.196.254:33023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:07:29.620051 15623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:07:29.620361 15623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:07:29.626156 15657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:29.626187 15660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:29.626230 15623 server_base.cc:1034] running on GCE node
W20260925 19:07:29.626255 15658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:29.626595 15623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:07:29.626948 15623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:07:29.628122 15623 hybrid_clock.cc:648] HybridClock initialized: now 1790363249628111 us; error 38 us; skew 500 ppm
Sep 25 19:07:29 dist-test-slave-rn98 krb5kdc[14481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:29.631255 15623 init.cc:377] Logged in from keytab as kudu/127.0.196.193@KRBTEST.COM (short username kudu)
I20260925 19:07:29.631892 15623 webserver.cc:460] Webserver started at http://127.0.196.193:40425/ using document root <none> and password file <none>
I20260925 19:07:29.632172 15623 fs_manager.cc:359] Metadata directory not provided
I20260925 19:07:29.632246 15623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:07:29.634013 15623 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 19:07:29.634905 15667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:29.635111 15623 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:07:29.635208 15623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "7c638a81e43148dc9b88b85642501874"
format_stamp: "Formatted at 2026-09-25 19:07:27 on dist-test-slave-rn98"
server_key: "8323b699a95b6ee25b7b8b60cc883d24"
server_key_iv: "be024b255af4912b7f144a4fb92b4f64"
server_key_version: "encryptionkey@0"
I20260925 19:07:29.635684 15623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:07:29.655224 15623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:07:29.657697 15623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:07:29.657819 15623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:07:29.658149 15623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:07:29.658687 15674 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 19:07:29.660719 15623 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 19:07:29.660797 15623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 19:07:29.660854 15623 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 19:07:29.661554 15623 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 19:07:29.661618 15623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 19:07:29.661675 15674 tablet_bootstrap.cc:492] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874: Bootstrap starting.
I20260925 19:07:29.669167 15623 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:44299
I20260925 19:07:29.669209 15781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:44299 every 8 connection(s)
I20260925 19:07:29.669641 15623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
I20260925 19:07:29.672943 15674 log.cc:824] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874: Log is configured to *not* fsync() on all Append() calls
I20260925 19:07:29.675534 15674 tablet_bootstrap.cc:492] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:07:29.675868 15674 tablet_bootstrap.cc:492] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874: Bootstrap complete.
I20260925 19:07:29.677191 15674 ts_tablet_manager.cc:1397] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.008s
I20260925 19:07:29.678792 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 15623
I20260925 19:07:29.679069 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:42299
--local_ip_for_outbound_sockets=127.0.196.194
--tserver_master_addrs=127.0.196.254:33023
--webserver_port=32849
--webserver_interface=127.0.196.194
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:39553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:07:29.678875 15674 raft_consensus.cc:348] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } }
I20260925 19:07:29.679224 15674 raft_consensus.cc:729] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c638a81e43148dc9b88b85642501874, State: Initialized, Role: FOLLOWER
I20260925 19:07:29.679396 15674 consensus_queue.cc:260] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } }
I20260925 19:07:29.679701 15674 ts_tablet_manager.cc:1428] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260925 19:07:29.765592 15784 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:07:29.670481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.193:49131)
0925 19:07:29.670706 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:29.670751 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:29.671238 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:29.765366 (+ 94128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":109,"threads_started":1}
W20260925 19:07:29.765889 15782 heartbeater.cc:644] Failed to heartbeat to 127.0.196.254:33023 (0 consecutive failures): Network error: Failed to ping master at 127.0.196.254:33023: Client connection negotiation failed: client connection to 127.0.196.254:33023: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 19:07:29.766242 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:29.766122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.193:46005)
0925 19:07:29.766178 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:29.766223 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 19:07:29.766583 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:29.766483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.193:46161)
0925 19:07:29.766535 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:29.766566 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 19:07:29.766707 15782 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 19:07:29.788511 15787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:07:29.788802 15787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:07:29.788848 15787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:07:29.790130 15787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:07:29.790220 15787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:07:29.790263 15787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:07:29.792088 15787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:39553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:42299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=32849
--tserver_master_addrs=127.0.196.254:33023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:07:29.792412 15787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:07:29.792676 15787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:07:29.798534 15795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:29.798575 15787 server_base.cc:1034] running on GCE node
W20260925 19:07:29.798539 15792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:29.798539 15793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:29.799057 15787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:07:29.799327 15787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:07:29.800486 15787 hybrid_clock.cc:648] HybridClock initialized: now 1790363249800438 us; error 65 us; skew 500 ppm
Sep 25 19:07:29 dist-test-slave-rn98 krb5kdc[14481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:29.803123 15787 init.cc:377] Logged in from keytab as kudu/127.0.196.194@KRBTEST.COM (short username kudu)
I20260925 19:07:29.803606 15787 webserver.cc:460] Webserver started at http://127.0.196.194:32849/ using document root <none> and password file <none>
I20260925 19:07:29.803845 15787 fs_manager.cc:359] Metadata directory not provided
I20260925 19:07:29.803917 15787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:07:29.805315 15787 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:07:29.805939 15802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:29.806129 15787 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:07:29.806229 15787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "1eb37032863941f18f46907b9d7a07dc"
format_stamp: "Formatted at 2026-09-25 19:07:27 on dist-test-slave-rn98"
server_key: "48057dd7b0d0d70f9689ef5c7c3b2398"
server_key_iv: "81c872f7cffceb5bcc9a6dc2fa8e1f2d"
server_key_version: "encryptionkey@0"
I20260925 19:07:29.806612 15787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:07:29.819954 15787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:07:29.822047 15787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:07:29.822223 15787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:07:29.822500 15787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:07:29.822861 15787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:07:29.822917 15787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:29.822961 15787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:07:29.822988 15787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:29.829155 15787 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:42299
I20260925 19:07:29.829195 15915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:42299 every 8 connection(s)
I20260925 19:07:29.829571 15787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
I20260925 19:07:29.830829 15918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:29.830295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.194:52485)
0925 19:07:29.830484 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:29.830570 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":79,"threads_started":1}
W20260925 19:07:29.831249 15916 heartbeater.cc:644] Failed to heartbeat to 127.0.196.254:33023 (0 consecutive failures): Network error: Failed to ping master at 127.0.196.254:33023: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
I20260925 19:07:29.831549 15918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:29.831445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.194:38951)
0925 19:07:29.831497 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:29.831525 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 19:07:29.831832 15918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:29.831746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.194:48979)
0925 19:07:29.831790 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:29.831812 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 19:07:29.831939 15916 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 19:07:29.835729 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 15787
I20260925 19:07:29.835986 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:44467
--local_ip_for_outbound_sockets=127.0.196.195
--tserver_master_addrs=127.0.196.254:33023
--webserver_port=41353
--webserver_interface=127.0.196.195
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:39553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:07:29.920147 15919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:07:29.920428 15919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:07:29.920467 15919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:07:29.921686 15919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:07:29.921767 15919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:07:29.921813 15919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:07:29.923538 15919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:39553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:44467
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=41353
--tserver_master_addrs=127.0.196.254:33023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:07:29.923795 15919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:07:29.924055 15919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:07:29.929306 15924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:29.929313 15925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:29.929313 15927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:29.929593 15919 server_base.cc:1034] running on GCE node
I20260925 19:07:29.929785 15919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:07:29.930025 15919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:07:29.931164 15919 hybrid_clock.cc:648] HybridClock initialized: now 1790363249931153 us; error 29 us; skew 500 ppm
Sep 25 19:07:29 dist-test-slave-rn98 krb5kdc[14481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:29.933876 15919 init.cc:377] Logged in from keytab as kudu/127.0.196.195@KRBTEST.COM (short username kudu)
I20260925 19:07:29.934422 15919 webserver.cc:460] Webserver started at http://127.0.196.195:41353/ using document root <none> and password file <none>
I20260925 19:07:29.934777 15919 fs_manager.cc:359] Metadata directory not provided
I20260925 19:07:29.934828 15919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:07:29.936266 15919 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:07:29.937008 15934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:29.937234 15919 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:07:29.937327 15919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "04f233d4c02b413daf423a661674cd48"
format_stamp: "Formatted at 2026-09-25 19:07:27 on dist-test-slave-rn98"
server_key: "8ac3f86a1be2cd8a1a4d3cc5c89d3de8"
server_key_iv: "2e1b15cb0a9a9a80b0bd66d5e9420906"
server_key_version: "encryptionkey@0"
I20260925 19:07:29.937770 15919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:07:29.952739 15919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:07:29.955158 15919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:07:29.955289 15919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:07:29.955487 15919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:07:29.955762 15919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:07:29.955796 15919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:29.955819 15919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:07:29.955832 15919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:29.961620 15919 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:44467
I20260925 19:07:29.961707 16047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:44467 every 8 connection(s)
I20260925 19:07:29.962064 15919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
I20260925 19:07:29.963301 16050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:29.962860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.195:43933)
0925 19:07:29.963038 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:29.963153 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
W20260925 19:07:29.963719 16048 heartbeater.cc:644] Failed to heartbeat to 127.0.196.254:33023 (0 consecutive failures): Network error: Failed to ping master at 127.0.196.254:33023: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
I20260925 19:07:29.964022 16050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:29.963913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.195:45793)
0925 19:07:29.963964 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:29.963997 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 19:07:29.964295 16050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:29.964209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.195:46307)
0925 19:07:29.964254 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:29.964276 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 19:07:29.964404 16048 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 19:07:29.971508 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 15919
I20260925 19:07:30.767403 16052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:30.767151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.193:36057)
0925 19:07:30.767299 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:30.767342 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260925 19:07:30.832614 16053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:30.832330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.194:34931)
0925 19:07:30.832484 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:30.832531 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":60,"threads_started":1}
I20260925 19:07:30.964946 16054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:30.964716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.195:39521)
0925 19:07:30.964842 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:30.964882 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":57,"threads_started":1}
I20260925 19:07:31.324904 16055 raft_consensus.cc:482] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:07:31.325028 16055 raft_consensus.cc:504] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } }
I20260925 19:07:31.325399 16055 leader_election.cc:288] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1eb37032863941f18f46907b9d7a07dc (127.0.196.194:42299), 04f233d4c02b413daf423a661674cd48 (127.0.196.195:44467)
Sep 25 19:07:31 dist-test-slave-rn98 krb5kdc[14481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for kudu/127.0.196.194@KRBTEST.COM
Sep 25 19:07:31 dist-test-slave-rn98 krb5kdc[14481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for kudu/127.0.196.195@KRBTEST.COM
I20260925 19:07:31.337420 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:31.325940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:46369 (local address 127.0.196.195:44467)
0925 19:07:31.326065 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:31.326078 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:31.326111 (+ 33us) server_negotiation.cc:407] Connection header received
0925 19:07:31.326510 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:31.326512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:31.326548 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:31.326770 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:31.327469 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:31.328326 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:31.329380 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:31.329761 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:31.329842 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:31.333150 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:31.333155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:31.333155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:31.333276 (+ 121us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:31.335384 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:31.335974 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:31.335977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:31.335978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:31.336079 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:31.336383 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:31.336384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:31.336384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:31.336642 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:31.336734 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:31.337116 (+ 382us) server_negotiation.cc:299] Negotiation successful
0925 19:07:31.337273 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":48,"threads_started":1}
I20260925 19:07:31.337415 16058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:31.325804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:44467 (local address 127.0.196.193:46369)
0925 19:07:31.325901 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:31.325911 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:31.325953 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:31.326771 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:31.326772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:31.327106 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:07:31.327314 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:31.327320 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:31.328455 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:31.328456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:31.329263 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:31.329265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:31.329892 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:31.329894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:31.330042 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:31.330828 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:07:31.330836 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:07:31.333035 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:07:31.335482 (+ 2447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:31.335489 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:31.335490 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:31.335856 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:31.336157 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:31.336158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:31.336159 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:31.336305 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:31.336717 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:07:31.336723 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:07:31.336895 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 19:07:31.337049 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 19:07:31.337279 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":70,"thread_start_us":51,"threads_started":1}
I20260925 19:07:31.337589 16057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:31.325696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:58557 (local address 127.0.196.194:42299)
0925 19:07:31.325865 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:31.325878 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:31.325917 (+ 39us) server_negotiation.cc:407] Connection header received
0925 19:07:31.326378 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:31.326381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:31.326473 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:31.326600 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:31.327455 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:31.328318 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:31.329347 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:31.329761 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:31.329972 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:31.332978 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:31.332986 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:31.332986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:31.333007 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:31.335384 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:31.335947 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:31.335950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:31.335951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:31.336056 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:31.336371 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:31.336372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:31.336373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:31.336639 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:31.336730 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:31.337196 (+ 466us) server_negotiation.cc:299] Negotiation successful
0925 19:07:31.337385 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":52,"threads_started":1}
I20260925 19:07:31.338207 16056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:31.325595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:42299 (local address 127.0.196.193:58557)
0925 19:07:31.325718 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:31.325730 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:31.325818 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:31.326697 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:31.326700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:31.327114 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:07:31.327314 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:31.327320 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:31.328424 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:31.328425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:31.329262 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:31.329264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:31.329942 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:31.329943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:31.330034 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:31.330717 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:07:31.330732 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:07:31.332743 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:07:31.335495 (+ 2752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:31.335496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:31.335497 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:31.335852 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:31.336138 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:31.336139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:31.336139 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:31.336305 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:31.336720 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:07:31.336726 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:07:31.336849 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 19:07:31.337112 (+ 263us) client_negotiation.cc:240] Negotiation successful
0925 19:07:31.337380 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":118,"thread_start_us":83,"threads_started":1}
W20260925 19:07:31.338477 15668 leader_election.cc:341] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 04f233d4c02b413daf423a661674cd48 (127.0.196.195:44467): Not found: Tablet not found: c8e8fe384a3441e0b9285a8d86d142cf
W20260925 19:07:31.339061 15668 leader_election.cc:341] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1eb37032863941f18f46907b9d7a07dc (127.0.196.194:42299): Not found: Tablet not found: c8e8fe384a3441e0b9285a8d86d142cf
I20260925 19:07:31.339111 15668 leader_election.cc:302] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c638a81e43148dc9b88b85642501874; no voters: 04f233d4c02b413daf423a661674cd48, 1eb37032863941f18f46907b9d7a07dc
I20260925 19:07:31.339242 16055 raft_consensus.cc:2726] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 19:07:31.768105 16056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:31.767939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.193:60953)
0925 19:07:31.768017 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:31.768064 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 19:07:31.833375 16060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:31.833146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.194:39287)
0925 19:07:31.833277 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:31.833317 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":59,"threads_started":1}
I20260925 19:07:31.965790 16061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:31.965440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.195:37423)
0925 19:07:31.965705 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:31.965746 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":169,"threads_started":1}
I20260925 19:07:32.768935 16062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:32.768699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.193:39545)
0925 19:07:32.768839 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:32.768885 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260925 19:07:32.834144 16063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:32.833903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.194:55455)
0925 19:07:32.834050 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:32.834097 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":61,"threads_started":1}
I20260925 19:07:32.966584 16064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:32.966264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.195:40547)
0925 19:07:32.966481 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:32.966525 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":80,"threads_started":1}
I20260925 19:07:33.471982 16065 raft_consensus.cc:482] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:07:33.472102 16065 raft_consensus.cc:504] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } }
I20260925 19:07:33.472306 16065 leader_election.cc:288] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1eb37032863941f18f46907b9d7a07dc (127.0.196.194:42299), 04f233d4c02b413daf423a661674cd48 (127.0.196.195:44467)
W20260925 19:07:33.472695 15668 leader_election.cc:341] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1eb37032863941f18f46907b9d7a07dc (127.0.196.194:42299): Not found: Tablet not found: c8e8fe384a3441e0b9285a8d86d142cf
W20260925 19:07:33.472771 15668 leader_election.cc:341] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 04f233d4c02b413daf423a661674cd48 (127.0.196.195:44467): Not found: Tablet not found: c8e8fe384a3441e0b9285a8d86d142cf
I20260925 19:07:33.472790 15668 leader_election.cc:302] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c638a81e43148dc9b88b85642501874; no voters: 04f233d4c02b413daf423a661674cd48, 1eb37032863941f18f46907b9d7a07dc
I20260925 19:07:33.472862 16065 raft_consensus.cc:2726] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 19:07:33.769685 16066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:33.769431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.193:54873)
0925 19:07:33.769599 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:33.769638 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":68,"threads_started":1}
I20260925 19:07:33.834967 16067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:33.834718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.194:36301)
0925 19:07:33.834856 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:33.834901 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":57,"threads_started":1}
I20260925 19:07:33.967403 16068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:33.967178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.195:43171)
0925 19:07:33.967313 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:33.967357 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260925 19:07:34.770511 16069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:34.770217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.193:55523)
0925 19:07:34.770409 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:34.770461 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":78,"threads_started":1}
I20260925 19:07:34.835825 16070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:34.835546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.194:40635)
0925 19:07:34.835726 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:34.835775 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":78,"threads_started":1}
I20260925 19:07:34.968205 16071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:34.967966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.195:55103)
0925 19:07:34.968102 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:34.968136 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260925 19:07:35.682000 16072 raft_consensus.cc:482] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:07:35.682096 16072 raft_consensus.cc:504] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } }
I20260925 19:07:35.682348 16072 leader_election.cc:288] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1eb37032863941f18f46907b9d7a07dc (127.0.196.194:42299), 04f233d4c02b413daf423a661674cd48 (127.0.196.195:44467)
W20260925 19:07:35.682848 15668 leader_election.cc:341] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1eb37032863941f18f46907b9d7a07dc (127.0.196.194:42299): Not found: Tablet not found: c8e8fe384a3441e0b9285a8d86d142cf
W20260925 19:07:35.682924 15668 leader_election.cc:341] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 04f233d4c02b413daf423a661674cd48 (127.0.196.195:44467): Not found: Tablet not found: c8e8fe384a3441e0b9285a8d86d142cf
I20260925 19:07:35.682938 15668 leader_election.cc:302] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c638a81e43148dc9b88b85642501874; no voters: 04f233d4c02b413daf423a661674cd48, 1eb37032863941f18f46907b9d7a07dc
I20260925 19:07:35.683048 16072 raft_consensus.cc:2726] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 19:07:35.771252 16073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:35.771021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.193:43159)
0925 19:07:35.771160 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:35.771203 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
I20260925 19:07:35.836777 16074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:35.836492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.194:59575)
0925 19:07:35.836655 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:35.836728 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":68,"threads_started":1}
I20260925 19:07:35.969084 16075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:35.968778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.195:57609)
0925 19:07:35.968982 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:35.969032 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":95,"threads_started":1}
I20260925 19:07:36.772089 16076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:36.771816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.193:56363)
0925 19:07:36.771996 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:36.772042 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":88,"threads_started":1}
I20260925 19:07:36.837661 16077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:36.837386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.194:35059)
0925 19:07:36.837561 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:36.837610 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":69,"threads_started":1}
I20260925 19:07:36.969986 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:36.969734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.195:38063)
0925 19:07:36.969895 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:36.969937 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":70,"threads_started":1}
I20260925 19:07:37.772944 16079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:37.772721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.193:34211)
0925 19:07:37.772848 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:37.772894 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260925 19:07:37.838732 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:37.838293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.194:54763)
0925 19:07:37.838533 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:37.838587 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":111,"threads_started":1}
I20260925 19:07:37.924896 16081 raft_consensus.cc:482] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:07:37.925021 16081 raft_consensus.cc:504] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } }
I20260925 19:07:37.925279 16081 leader_election.cc:288] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1eb37032863941f18f46907b9d7a07dc (127.0.196.194:42299), 04f233d4c02b413daf423a661674cd48 (127.0.196.195:44467)
W20260925 19:07:37.925756 15668 leader_election.cc:341] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 04f233d4c02b413daf423a661674cd48 (127.0.196.195:44467): Not found: Tablet not found: c8e8fe384a3441e0b9285a8d86d142cf
W20260925 19:07:37.925823 15668 leader_election.cc:341] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1eb37032863941f18f46907b9d7a07dc (127.0.196.194:42299): Not found: Tablet not found: c8e8fe384a3441e0b9285a8d86d142cf
I20260925 19:07:37.925839 15668 leader_election.cc:302] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c638a81e43148dc9b88b85642501874; no voters: 04f233d4c02b413daf423a661674cd48, 1eb37032863941f18f46907b9d7a07dc
I20260925 19:07:37.925967 16081 raft_consensus.cc:2726] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 19:07:37.970882 16082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:37.970577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.195:43007)
0925 19:07:37.970776 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:37.970818 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":132,"threads_started":1}
I20260925 19:07:38.773782 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:38.773518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.193:42301)
0925 19:07:38.773689 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:38.773733 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":103,"threads_started":1}
I20260925 19:07:38.839427 16084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:38.839190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.194:58377)
0925 19:07:38.839333 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:38.839378 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":60,"threads_started":1}
I20260925 19:07:38.971727 16085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:38.971464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.195:46977)
0925 19:07:38.971630 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:38.971678 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":73,"threads_started":1}
I20260925 19:07:39.774703 16086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:39.774417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.193:35709)
0925 19:07:39.774581 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:39.774654 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":65,"threads_started":1}
I20260925 19:07:39.840139 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:39.839933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.194:50131)
0925 19:07:39.840054 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:39.840090 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260925 19:07:39.972572 16088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:39.972278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.195:54941)
0925 19:07:39.972475 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:39.972522 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":113,"threads_started":1}
I20260925 19:07:40.775377 16089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:40.775179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.193:51261)
0925 19:07:40.775292 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:40.775330 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260925 19:07:40.841001 16090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:40.840749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.194:59293)
0925 19:07:40.840886 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:40.840947 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
I20260925 19:07:40.973512 16091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:40.973189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.195:60213)
0925 19:07:40.973415 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:40.973462 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":81,"threads_started":1}
I20260925 19:07:41.776186 16092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:41.775925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.193:32931)
0925 19:07:41.776088 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:41.776136 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":71,"threads_started":1}
I20260925 19:07:41.817346 16093 raft_consensus.cc:482] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:07:41.817448 16093 raft_consensus.cc:504] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb37032863941f18f46907b9d7a07dc" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42299 } } peers { permanent_uuid: "04f233d4c02b413daf423a661674cd48" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44467 } } peers { permanent_uuid: "7c638a81e43148dc9b88b85642501874" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 44299 } }
I20260925 19:07:41.817596 16093 leader_election.cc:288] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1eb37032863941f18f46907b9d7a07dc (127.0.196.194:42299), 04f233d4c02b413daf423a661674cd48 (127.0.196.195:44467)
W20260925 19:07:41.818033 15668 leader_election.cc:341] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 04f233d4c02b413daf423a661674cd48 (127.0.196.195:44467): Not found: Tablet not found: c8e8fe384a3441e0b9285a8d86d142cf
W20260925 19:07:41.818136 15668 leader_election.cc:341] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1eb37032863941f18f46907b9d7a07dc (127.0.196.194:42299): Not found: Tablet not found: c8e8fe384a3441e0b9285a8d86d142cf
I20260925 19:07:41.818176 15668 leader_election.cc:302] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c638a81e43148dc9b88b85642501874; no voters: 04f233d4c02b413daf423a661674cd48, 1eb37032863941f18f46907b9d7a07dc
I20260925 19:07:41.818270 16093 raft_consensus.cc:2726] T c8e8fe384a3441e0b9285a8d86d142cf P 7c638a81e43148dc9b88b85642501874 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 19:07:41.841867 16094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:41.841641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.194:38313)
0925 19:07:41.841771 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:41.841819 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
I20260925 19:07:41.974308 16095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:41.974070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.195:53685)
0925 19:07:41.974200 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:41.974242 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
I20260925 19:07:42.777052 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:42.776780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.193:51397)
0925 19:07:42.776949 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:42.777006 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":67,"threads_started":1}
I20260925 19:07:42.842706 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:42.842394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.194:49757)
0925 19:07:42.842536 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:42.842584 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":62,"threads_started":1}
I20260925 19:07:42.975144 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:42.974931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.195:38951)
0925 19:07:42.975057 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:42.975098 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260925 19:07:43.778038 16099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:43.777740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.193:56075)
0925 19:07:43.777893 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:43.777982 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":72,"threads_started":1}
I20260925 19:07:43.843477 16100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:43.843199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.194:36925)
0925 19:07:43.843360 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:43.843421 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":66,"threads_started":1}
I20260925 19:07:43.975970 16101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:43.975727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.195:55031)
0925 19:07:43.975868 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:43.975918 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260925 19:07:44.779047 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:44.778808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.193:60843)
0925 19:07:44.778949 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:44.778998 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":61,"threads_started":1}
I20260925 19:07:44.844364 16103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:44.844104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.194:58063)
0925 19:07:44.844246 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:44.844295 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":61,"threads_started":1}
I20260925 19:07:44.973536 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 15623
I20260925 19:07:44.976975 16104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:44.976697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33023 (local address 127.0.196.195:58445)
0925 19:07:44.976876 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:44.976926 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":79,"threads_started":1}
I20260925 19:07:44.979022 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 15787
I20260925 19:07:44.983698 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 15919
2026-09-25T19:07:44Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18880 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:07:45 dist-test-slave-rn98 krb5kdc[16107](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:07:45 dist-test-slave-rn98 krb5kdc[16107](info): set up 2 sockets
Sep 25 19:07:45 dist-test-slave-rn98 krb5kdc[16107](info): commencing operation
krb5kdc: starting...
W20260925 19:07:46.059360 787 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:07:46 dist-test-slave-rn98 krb5kdc[16107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:07:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:07:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:07:46.190591 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:41039
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:37989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:41039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:07:46.274334 16120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:07:46.274623 16120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:07:46.274696 16120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:07:46.276005 16120 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 19:07:46.276077 16120 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 19:07:46.276096 16120 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 19:07:46.276114 16120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:07:46.276149 16120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:07:46.276180 16120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:07:46.276203 16120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:07:46.276230 16120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:07:46.277864 16120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:37989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:41039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:41039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:07:46.278180 16120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:07:46.278434 16120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:07:46.284080 16126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:46.284118 16120 server_base.cc:1034] running on GCE node
W20260925 19:07:46.284083 16128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:46.284129 16125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:46.284668 16120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:07:46.284968 16120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:07:46.286144 16120 hybrid_clock.cc:648] HybridClock initialized: now 1790363266286115 us; error 42 us; skew 500 ppm
Sep 25 19:07:46 dist-test-slave-rn98 krb5kdc[16107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:46.289031 16120 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:07:46.289579 16120 webserver.cc:460] Webserver started at http://127.0.196.254:35695/ using document root <none> and password file <none>
I20260925 19:07:46.289845 16120 fs_manager.cc:359] Metadata directory not provided
I20260925 19:07:46.289917 16120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:07:46.290074 16120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:07:46.291337 16120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "283208106c934ffd993371388e6acb4c"
format_stamp: "Formatted at 2026-09-25 19:07:46 on dist-test-slave-rn98"
server_key: "05906fd3fcc23ff6e80894adc84c17eb"
server_key_iv: "7be0c67088632fc1166fc2f6c2bda3f6"
server_key_version: "encryptionkey@0"
I20260925 19:07:46.291726 16120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "283208106c934ffd993371388e6acb4c"
format_stamp: "Formatted at 2026-09-25 19:07:46 on dist-test-slave-rn98"
server_key: "05906fd3fcc23ff6e80894adc84c17eb"
server_key_iv: "7be0c67088632fc1166fc2f6c2bda3f6"
server_key_version: "encryptionkey@0"
I20260925 19:07:46.293206 16120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 19:07:46.294127 16135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:46.294391 16120 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:07:46.294497 16120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "283208106c934ffd993371388e6acb4c"
format_stamp: "Formatted at 2026-09-25 19:07:46 on dist-test-slave-rn98"
server_key: "05906fd3fcc23ff6e80894adc84c17eb"
server_key_iv: "7be0c67088632fc1166fc2f6c2bda3f6"
server_key_version: "encryptionkey@0"
I20260925 19:07:46.294593 16120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:07:46.305051 16120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:07:46.307252 16120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:07:46.307431 16120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:07:46.311782 16120 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:41039
I20260925 19:07:46.311849 16187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:41039 every 8 connection(s)
I20260925 19:07:46.312247 16120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:07:46.313210 16188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:07:46.315912 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 16120
I20260925 19:07:46.316006 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:07:46.316218 787 external_mini_cluster.cc:1428] Setting key 2fba45f9d6e815dcc222be87e2663dc1
I20260925 19:07:46.316452 16188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c: Bootstrap starting.
I20260925 19:07:46.317263 16188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c: Neither blocks nor log segments found. Creating new log.
I20260925 19:07:46.317581 16188 log.cc:824] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c: Log is configured to *not* fsync() on all Append() calls
I20260925 19:07:46.318451 16188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c: No bootstrap required, opened a new log
I20260925 19:07:46.320657 16188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283208106c934ffd993371388e6acb4c" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41039 } }
I20260925 19:07:46.320860 16188 raft_consensus.cc:374] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:07:46.320889 16188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283208106c934ffd993371388e6acb4c, State: Initialized, Role: FOLLOWER
I20260925 19:07:46.321027 16188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283208106c934ffd993371388e6acb4c" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41039 } }
I20260925 19:07:46.321117 16188 raft_consensus.cc:388] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:07:46.321163 16188 raft_consensus.cc:482] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:07:46.321235 16188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:07:46.322062 16188 raft_consensus.cc:504] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283208106c934ffd993371388e6acb4c" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41039 } }
I20260925 19:07:46.322189 16188 leader_election.cc:302] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 283208106c934ffd993371388e6acb4c; no voters:
I20260925 19:07:46.322384 16188 leader_election.cc:288] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:07:46.322430 16193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:07:46.322590 16193 raft_consensus.cc:686] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [term 1 LEADER]: Becoming Leader. State: Replica: 283208106c934ffd993371388e6acb4c, State: Running, Role: LEADER
I20260925 19:07:46.322729 16188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:07:46.322753 16193 consensus_queue.cc:237] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283208106c934ffd993371388e6acb4c" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41039 } }
I20260925 19:07:46.323238 16194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "283208106c934ffd993371388e6acb4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283208106c934ffd993371388e6acb4c" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41039 } } }
I20260925 19:07:46.323319 16194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [sys.catalog]: This master's current role is: LEADER
I20260925 19:07:46.323487 16195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 283208106c934ffd993371388e6acb4c. Latest consensus state: current_term: 1 leader_uuid: "283208106c934ffd993371388e6acb4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283208106c934ffd993371388e6acb4c" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41039 } } }
I20260925 19:07:46.323544 16195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [sys.catalog]: This master's current role is: LEADER
Sep 25 19:07:46 dist-test-slave-rn98 krb5kdc[16107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
W20260925 19:07:46.324746 16209 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:07:46.324810 16209 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:07:46.324868 16199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:07:46.325048 16199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:07:46.327419 16199 catalog_manager.cc:1347] Generated new cluster ID: 3a354f1db28945af8e5b6ed8ba2b7cb0
I20260925 19:07:46.327495 16199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:07:46.331660 16191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:46.316803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53484 (local address 127.0.196.254:41039)
0925 19:07:46.316993 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:46.317003 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:46.317041 (+ 38us) server_negotiation.cc:407] Connection header received
0925 19:07:46.317548 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:46.317556 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:46.317609 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:46.317781 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:46.318581 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:46.319557 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:46.320741 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:46.321469 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:46.321556 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:46.324732 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:46.324736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:46.324737 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:46.324750 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:46.327116 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:46.327588 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:46.327592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:46.327592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:46.327701 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:46.327922 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:46.327923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:46.327924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:46.328226 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:46.330738 (+ 2512us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:46.331145 (+ 407us) server_negotiation.cc:299] Negotiation successful
0925 19:07:46.331392 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":105,"threads_started":1}
I20260925 19:07:46.341675 16199 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:07:46.342252 16199 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:07:46.344914 16199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c: Generated new TSK 0
I20260925 19:07:46.345120 16199 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
I20260925 19:07:46.375842 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:0
--local_ip_for_outbound_sockets=127.0.196.193
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:41039
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:37989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:07:46.480919 16214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:07:46.481204 16214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:07:46.481253 16214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:07:46.483083 16214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:07:46.483170 16214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:07:46.483222 16214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:07:46.484990 16214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:37989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:41039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:07:46.485314 16214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:07:46.485576 16214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:07:46.491235 16220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:46.491189 16219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:46.491235 16222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:46.491287 16214 server_base.cc:1034] running on GCE node
I20260925 19:07:46.491717 16214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:07:46.491956 16214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:07:46.493168 16214 hybrid_clock.cc:648] HybridClock initialized: now 1790363266493161 us; error 147 us; skew 500 ppm
Sep 25 19:07:46 dist-test-slave-rn98 krb5kdc[16107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:46.496162 16214 init.cc:377] Logged in from keytab as kudu/127.0.196.193@KRBTEST.COM (short username kudu)
I20260925 19:07:46.496711 16214 webserver.cc:460] Webserver started at http://127.0.196.193:34401/ using document root <none> and password file <none>
I20260925 19:07:46.496975 16214 fs_manager.cc:359] Metadata directory not provided
I20260925 19:07:46.497051 16214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:07:46.497207 16214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:07:46.498410 16214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/data/instance:
uuid: "32b8f17051084c27a6ef472c2fbb312e"
format_stamp: "Formatted at 2026-09-25 19:07:46 on dist-test-slave-rn98"
server_key: "95bb34c2589216a0420e9cdce5fb14cd"
server_key_iv: "9e4d076f09bf6a566d12e28957e4dd2f"
server_key_version: "encryptionkey@0"
I20260925 19:07:46.498906 16214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/wal/instance:
uuid: "32b8f17051084c27a6ef472c2fbb312e"
format_stamp: "Formatted at 2026-09-25 19:07:46 on dist-test-slave-rn98"
server_key: "95bb34c2589216a0420e9cdce5fb14cd"
server_key_iv: "9e4d076f09bf6a566d12e28957e4dd2f"
server_key_version: "encryptionkey@0"
I20260925 19:07:46.500504 16214 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 19:07:46.501338 16229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:46.502056 16214 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:07:46.502202 16214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "32b8f17051084c27a6ef472c2fbb312e"
format_stamp: "Formatted at 2026-09-25 19:07:46 on dist-test-slave-rn98"
server_key: "95bb34c2589216a0420e9cdce5fb14cd"
server_key_iv: "9e4d076f09bf6a566d12e28957e4dd2f"
server_key_version: "encryptionkey@0"
I20260925 19:07:46.502295 16214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:07:46.522145 16214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:07:46.525012 16214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:07:46.525138 16214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:07:46.525333 16214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:07:46.525696 16214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:07:46.525756 16214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:46.525801 16214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:07:46.525836 16214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:46.531958 16214 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:35643
I20260925 19:07:46.532055 16342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:35643 every 8 connection(s)
I20260925 19:07:46.532414 16214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
Sep 25 19:07:46 dist-test-slave-rn98 krb5kdc[16107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:07:46.542338 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 16214
I20260925 19:07:46.542451 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/wal/instance
I20260925 19:07:46.542745 787 external_mini_cluster.cc:1428] Setting key bf911ee872b83c8a6824b6f6cfd13ee7
I20260925 19:07:46.542817 16191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:46.533186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:32951 (local address 127.0.196.254:41039)
0925 19:07:46.533258 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:46.533260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:46.533511 (+ 251us) server_negotiation.cc:407] Connection header received
0925 19:07:46.534084 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:46.534086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:46.534099 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:46.534149 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:46.535288 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:46.535803 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:46.536871 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:46.537110 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:46.537196 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:46.539748 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:46.539750 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:46.539750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:46.539765 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:46.541059 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:46.541523 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:46.541524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:46.541524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:46.541587 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:46.541966 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:46.541967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:46.541967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:46.542135 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:46.542219 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:46.542600 (+ 381us) server_negotiation.cc:299] Negotiation successful
0925 19:07:46.542723 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 19:07:46.543175 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:46.533162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.193:32951)
0925 19:07:46.533356 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:46.533397 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:46.533957 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:46.534255 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:46.534259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:46.534948 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:07:46.535158 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:46.535164 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:46.535910 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:46.535911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:46.536759 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:46.536762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:46.537207 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:46.537209 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:46.537352 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:46.537977 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:07:46.537986 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:07:46.539617 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:07:46.541140 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:46.541147 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:46.541149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:46.541418 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:46.541686 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:46.541687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:46.541688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:46.541856 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:46.542235 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:07:46.542242 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:07:46.542393 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 19:07:46.542553 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 19:07:46.542768 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":93,"threads_started":1}
I20260925 19:07:46.543766 16343 heartbeater.cc:347] Connected to a master server at 127.0.196.254:41039
I20260925 19:07:46.543900 16343 heartbeater.cc:464] Registering TS with master...
I20260925 19:07:46.544098 16343 heartbeater.cc:511] Master 127.0.196.254:41039 requested a full tablet report, sending...
I20260925 19:07:46.544700 16152 ts_manager.cc:194] Registered new tserver with Master: 32b8f17051084c27a6ef472c2fbb312e (127.0.196.193:35643)
I20260925 19:07:46.545588 16152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.193@KRBTEST.COM'} at 127.0.196.193:32951
WARNING: no policy specified for kudu/127.0.196.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
I20260925 19:07:46.573984 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:0
--local_ip_for_outbound_sockets=127.0.196.194
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:41039
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:37989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:07:46.659792 16348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:07:46.660130 16348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:07:46.660159 16348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:07:46.661448 16348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:07:46.661499 16348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:07:46.661530 16348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:07:46.663188 16348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:37989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:41039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:07:46.663427 16348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:07:46.663633 16348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:07:46.669056 16354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:46.669073 16353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:46.669229 16348 server_base.cc:1034] running on GCE node
W20260925 19:07:46.669093 16356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:46.669548 16348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:07:46.669832 16348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:07:46.671111 16348 hybrid_clock.cc:648] HybridClock initialized: now 1790363266671098 us; error 51 us; skew 500 ppm
Sep 25 19:07:46 dist-test-slave-rn98 krb5kdc[16107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:46.673897 16348 init.cc:377] Logged in from keytab as kudu/127.0.196.194@KRBTEST.COM (short username kudu)
I20260925 19:07:46.674484 16348 webserver.cc:460] Webserver started at http://127.0.196.194:37129/ using document root <none> and password file <none>
I20260925 19:07:46.674863 16348 fs_manager.cc:359] Metadata directory not provided
I20260925 19:07:46.674944 16348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:07:46.675094 16348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:07:46.676272 16348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/data/instance:
uuid: "e8b6377bb8db4f038c6ae369d9906e80"
format_stamp: "Formatted at 2026-09-25 19:07:46 on dist-test-slave-rn98"
server_key: "5ac6036f557c209eee2b864d9e8c137e"
server_key_iv: "be22d3fe09633d6c66857d29c19f172f"
server_key_version: "encryptionkey@0"
I20260925 19:07:46.676672 16348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/wal/instance:
uuid: "e8b6377bb8db4f038c6ae369d9906e80"
format_stamp: "Formatted at 2026-09-25 19:07:46 on dist-test-slave-rn98"
server_key: "5ac6036f557c209eee2b864d9e8c137e"
server_key_iv: "be22d3fe09633d6c66857d29c19f172f"
server_key_version: "encryptionkey@0"
I20260925 19:07:46.678231 16348 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:07:46.679412 16363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:46.679651 16348 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:07:46.679767 16348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "e8b6377bb8db4f038c6ae369d9906e80"
format_stamp: "Formatted at 2026-09-25 19:07:46 on dist-test-slave-rn98"
server_key: "5ac6036f557c209eee2b864d9e8c137e"
server_key_iv: "be22d3fe09633d6c66857d29c19f172f"
server_key_version: "encryptionkey@0"
I20260925 19:07:46.679868 16348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:07:46.702607 16348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:07:46.705483 16348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:07:46.705626 16348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:07:46.705849 16348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:07:46.706140 16348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:07:46.706173 16348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:46.706199 16348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:07:46.706246 16348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:46.712963 16348 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:43989
I20260925 19:07:46.713043 16476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:43989 every 8 connection(s)
I20260925 19:07:46.713397 16348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
I20260925 19:07:46.720140 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 16348
Sep 25 19:07:46 dist-test-slave-rn98 krb5kdc[16107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:07:46.720281 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/wal/instance
I20260925 19:07:46.720535 787 external_mini_cluster.cc:1428] Setting key 70ec29457f560ab4c401ac67b4a63954
I20260925 19:07:46.724833 16191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:46.714296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:42585 (local address 127.0.196.254:41039)
0925 19:07:46.714381 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:46.714383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:46.714678 (+ 295us) server_negotiation.cc:407] Connection header received
0925 19:07:46.715191 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:46.715192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:46.715204 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:46.715264 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:46.716274 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:46.716850 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:46.717993 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:46.718223 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:46.718287 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:46.720915 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:46.720917 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:46.720917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:46.720926 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:46.722844 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:46.723421 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:46.723423 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:46.723424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:46.723508 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:46.723914 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:46.723916 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:46.723916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:46.724075 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:46.724135 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:46.724579 (+ 444us) server_negotiation.cc:299] Negotiation successful
0925 19:07:46.724681 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 19:07:46.724874 16479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:46.714292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.194:42585)
0925 19:07:46.714495 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:46.714535 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:46.715095 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:46.715340 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:46.715343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:46.715964 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:07:46.716175 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:46.716180 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:46.717040 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:46.717041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:46.717899 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:46.717901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:46.718322 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:46.718323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:46.718446 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:46.719123 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:07:46.719132 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:07:46.720791 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:07:46.722952 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:46.722964 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:46.722968 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:46.723298 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:46.723648 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:46.723649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:46.723650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:46.723836 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:46.724175 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:07:46.724183 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:07:46.724410 (+ 227us) client_negotiation.cc:769] Sending connection context
0925 19:07:46.724529 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 19:07:46.724683 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":103,"threads_started":1}
I20260925 19:07:46.725803 16477 heartbeater.cc:347] Connected to a master server at 127.0.196.254:41039
I20260925 19:07:46.725935 16477 heartbeater.cc:464] Registering TS with master...
I20260925 19:07:46.726156 16477 heartbeater.cc:511] Master 127.0.196.254:41039 requested a full tablet report, sending...
I20260925 19:07:46.726658 16152 ts_manager.cc:194] Registered new tserver with Master: e8b6377bb8db4f038c6ae369d9906e80 (127.0.196.194:43989)
I20260925 19:07:46.727080 16152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.194@KRBTEST.COM'} at 127.0.196.194:42585
WARNING: no policy specified for kudu/127.0.196.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
I20260925 19:07:46.750828 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:0
--local_ip_for_outbound_sockets=127.0.196.195
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:41039
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:37989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:07:46.841300 16482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:07:46.841554 16482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:07:46.841584 16482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:07:46.843086 16482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:07:46.843145 16482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:07:46.843190 16482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:07:46.844967 16482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:37989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:41039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:07:46.845286 16482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:07:46.845579 16482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:07:46.851528 16488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:46.851673 16490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:07:46.851524 16487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:07:46.851718 16482 server_base.cc:1034] running on GCE node
I20260925 19:07:46.852084 16482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:07:46.852612 16482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:07:46.853945 16482 hybrid_clock.cc:648] HybridClock initialized: now 1790363266853931 us; error 35 us; skew 500 ppm
Sep 25 19:07:46 dist-test-slave-rn98 krb5kdc[16107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:07:46.857043 16482 init.cc:377] Logged in from keytab as kudu/127.0.196.195@KRBTEST.COM (short username kudu)
I20260925 19:07:46.857699 16482 webserver.cc:460] Webserver started at http://127.0.196.195:40171/ using document root <none> and password file <none>
I20260925 19:07:46.858001 16482 fs_manager.cc:359] Metadata directory not provided
I20260925 19:07:46.858081 16482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:07:46.858328 16482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:07:46.859683 16482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/data/instance:
uuid: "b8ebe057555540bf8d65ba18ee36a54f"
format_stamp: "Formatted at 2026-09-25 19:07:46 on dist-test-slave-rn98"
server_key: "f5c943bf10f5c61d4cbc853dc01c4209"
server_key_iv: "bf0013e28242eb59c07c7651e107d25d"
server_key_version: "encryptionkey@0"
I20260925 19:07:46.860116 16482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/wal/instance:
uuid: "b8ebe057555540bf8d65ba18ee36a54f"
format_stamp: "Formatted at 2026-09-25 19:07:46 on dist-test-slave-rn98"
server_key: "f5c943bf10f5c61d4cbc853dc01c4209"
server_key_iv: "bf0013e28242eb59c07c7651e107d25d"
server_key_version: "encryptionkey@0"
I20260925 19:07:46.861871 16482 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:07:46.862838 16497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:46.863152 16482 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:07:46.863263 16482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "b8ebe057555540bf8d65ba18ee36a54f"
format_stamp: "Formatted at 2026-09-25 19:07:46 on dist-test-slave-rn98"
server_key: "f5c943bf10f5c61d4cbc853dc01c4209"
server_key_iv: "bf0013e28242eb59c07c7651e107d25d"
server_key_version: "encryptionkey@0"
I20260925 19:07:46.863339 16482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:07:46.891441 16482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:07:46.894068 16482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:07:46.894281 16482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:07:46.894580 16482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:07:46.894968 16482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:07:46.895030 16482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:46.895074 16482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:07:46.895100 16482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:07:46.901576 16482 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:43139
I20260925 19:07:46.901610 16610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:43139 every 8 connection(s)
I20260925 19:07:46.901978 16482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
I20260925 19:07:46.906762 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 16482
I20260925 19:07:46.906863 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/wal/instance
I20260925 19:07:46.907056 787 external_mini_cluster.cc:1428] Setting key dfe369953adfec376696af17ea366823
Sep 25 19:07:46 dist-test-slave-rn98 krb5kdc[16107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:07:46.912647 16191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:46.902802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:49533 (local address 127.0.196.254:41039)
0925 19:07:46.902873 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:46.902875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:46.903121 (+ 246us) server_negotiation.cc:407] Connection header received
0925 19:07:46.903632 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:46.903633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:46.903643 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:46.903698 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:46.904799 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:46.905371 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:46.906592 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:46.906948 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:46.907039 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:46.909813 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:46.909815 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:46.909815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:46.909822 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:46.911237 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:46.911692 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:46.911693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:46.911693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:46.911747 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:46.912026 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:46.912027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:46.912027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:46.912145 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:46.912202 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:46.912510 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 19:07:46.912578 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 19:07:46.912928 16613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:46.902811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.195:49533)
0925 19:07:46.902978 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:46.903021 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:46.903535 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:46.903760 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:46.903763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:46.904471 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:07:46.904689 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:46.904695 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:46.905462 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:46.905463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:46.906494 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:46.906497 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:46.907044 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:46.907045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:46.907216 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:46.908060 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:07:46.908072 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:07:46.909716 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:07:46.911356 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:46.911363 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:46.911366 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:46.911623 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:46.911813 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:07:46.911813 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:07:46.911814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:07:46.911937 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:07:46.912211 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:07:46.912216 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:07:46.912380 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 19:07:46.912481 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 19:07:46.912623 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260925 19:07:46.913520 16611 heartbeater.cc:347] Connected to a master server at 127.0.196.254:41039
I20260925 19:07:46.913641 16611 heartbeater.cc:464] Registering TS with master...
I20260925 19:07:46.913859 16611 heartbeater.cc:511] Master 127.0.196.254:41039 requested a full tablet report, sending...
I20260925 19:07:46.914325 16152 ts_manager.cc:194] Registered new tserver with Master: b8ebe057555540bf8d65ba18ee36a54f (127.0.196.195:43139)
I20260925 19:07:46.914798 16152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.195@KRBTEST.COM'} at 127.0.196.195:49533
I20260925 19:07:46.919306 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:07:46.928251 16191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:46.920783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53494 (local address 127.0.196.254:41039)
0925 19:07:46.920835 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:46.920838 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:46.920899 (+ 61us) server_negotiation.cc:407] Connection header received
0925 19:07:46.920953 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:46.920954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:46.920964 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:46.921023 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:07:46.921618 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:46.922179 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:46.923064 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:46.923347 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:46.923448 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:46.925133 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:07:46.925135 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:07:46.925135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:07:46.925146 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:07:46.926890 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:46.927222 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:46.927223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:46.927223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:46.927277 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:07:46.927619 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:07:46.927620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:07:46.927620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:07:46.927809 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:07:46.927912 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:46.928065 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 19:07:46.928138 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 19:07:46.930373 16152 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:07:46.932170 16152 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:07:46.940052 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:46.935228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52494 (local address 127.0.196.194:43989)
0925 19:07:46.935455 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:46.935462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:46.935477 (+ 15us) server_negotiation.cc:407] Connection header received
0925 19:07:46.935619 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:46.935623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:46.935642 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:46.935733 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:07:46.936009 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:46.937299 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:46.938977 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:46.939605 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:46.939674 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:46.939764 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:46.939910 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 19:07:46.939960 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":99,"threads_started":1}
I20260925 19:07:46.940042 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:46.935009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:43989 (local address 127.0.0.1:52494)
0925 19:07:46.935342 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:46.935391 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:46.935473 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:46.935847 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:46.935850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:46.935852 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:07:46.935918 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:46.935920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:46.937443 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:46.937444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:46.938879 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:46.938883 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:46.939699 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:46.939701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:46.939791 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:46.939817 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 19:07:46.939890 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 19:07:46.939952 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":63,"thread_start_us":61,"threads_started":1}
I20260925 19:07:46.940448 16626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:46.935415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43996 (local address 127.0.196.193:35643)
0925 19:07:46.935579 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:46.935583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:46.936292 (+ 709us) server_negotiation.cc:407] Connection header received
0925 19:07:46.936357 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:46.936359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:46.936377 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:46.936452 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:07:46.936694 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:46.937952 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:46.939114 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:46.939634 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:46.939731 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:46.939793 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:46.940299 (+ 506us) server_negotiation.cc:299] Negotiation successful
0925 19:07:46.940354 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
I20260925 19:07:46.940665 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:46.935023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:35643 (local address 127.0.0.1:43996)
0925 19:07:46.936183 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:46.936193 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:46.936253 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:46.936545 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:46.936546 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:46.936549 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:07:46.936601 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:46.936602 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:46.938075 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:46.938076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:46.939024 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:46.939026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:46.939758 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:46.939759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:46.939828 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:46.939833 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:07:46.939902 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 19:07:46.940579 (+ 677us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1143,"thread_start_us":124,"threads_started":1}
I20260925 19:07:46.940999 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:46.935009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:43139 (local address 127.0.0.1:50436)
0925 19:07:46.936045 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:46.936058 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:46.936116 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:46.936404 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:46.936405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:46.936407 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:07:46.936458 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:46.936459 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:46.937363 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:46.937364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:46.938880 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:46.938883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:46.940749 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:46.940751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:46.940817 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:46.940821 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:07:46.940887 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 19:07:46.940932 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"mutex_wait_us":77,"thread_start_us":98,"threads_started":1}
I20260925 19:07:46.941066 16627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:46.935448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50436 (local address 127.0.196.195:43139)
0925 19:07:46.935726 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:46.935731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:46.936162 (+ 431us) server_negotiation.cc:407] Connection header received
0925 19:07:46.936209 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:46.936212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:46.936226 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:46.936414 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:07:46.936505 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:46.937251 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:46.939807 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:46.940653 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:46.940748 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:46.940821 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:46.940904 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 19:07:46.940964 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":94,"threads_started":1}
I20260925 19:07:46.941183 16411 tablet_service.cc:1467] Processing CreateTablet for tablet f5c7140d4c4c4482a881068379ceb2cb (DEFAULT_TABLE table=test-table [id=fea2be3a872542488958f2ad20ac82f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:07:46.941566 16411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5c7140d4c4c4482a881068379ceb2cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:07:46.941679 16277 tablet_service.cc:1467] Processing CreateTablet for tablet f5c7140d4c4c4482a881068379ceb2cb (DEFAULT_TABLE table=test-table [id=fea2be3a872542488958f2ad20ac82f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:07:46.941998 16277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5c7140d4c4c4482a881068379ceb2cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:07:46.945300 16628 tablet_bootstrap.cc:492] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80: Bootstrap starting.
I20260925 19:07:46.946033 16628 tablet_bootstrap.cc:654] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80: Neither blocks nor log segments found. Creating new log.
I20260925 19:07:46.946033 16545 tablet_service.cc:1467] Processing CreateTablet for tablet f5c7140d4c4c4482a881068379ceb2cb (DEFAULT_TABLE table=test-table [id=fea2be3a872542488958f2ad20ac82f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:07:46.946375 16545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5c7140d4c4c4482a881068379ceb2cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:07:46.946724 16628 log.cc:824] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80: Log is configured to *not* fsync() on all Append() calls
I20260925 19:07:46.947146 16629 tablet_bootstrap.cc:492] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e: Bootstrap starting.
I20260925 19:07:46.947717 16629 tablet_bootstrap.cc:654] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e: Neither blocks nor log segments found. Creating new log.
I20260925 19:07:46.948155 16629 log.cc:824] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e: Log is configured to *not* fsync() on all Append() calls
I20260925 19:07:46.948717 16628 tablet_bootstrap.cc:492] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80: No bootstrap required, opened a new log
I20260925 19:07:46.948776 16628 ts_tablet_manager.cc:1397] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260925 19:07:46.949275 16632 tablet_bootstrap.cc:492] T f5c7140d4c4c4482a881068379ceb2cb P b8ebe057555540bf8d65ba18ee36a54f: Bootstrap starting.
I20260925 19:07:46.950026 16632 tablet_bootstrap.cc:654] T f5c7140d4c4c4482a881068379ceb2cb P b8ebe057555540bf8d65ba18ee36a54f: Neither blocks nor log segments found. Creating new log.
I20260925 19:07:46.950637 16632 log.cc:824] T f5c7140d4c4c4482a881068379ceb2cb P b8ebe057555540bf8d65ba18ee36a54f: Log is configured to *not* fsync() on all Append() calls
I20260925 19:07:46.951025 16628 raft_consensus.cc:348] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:07:46.951221 16628 raft_consensus.cc:374] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:07:46.951274 16628 raft_consensus.cc:729] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8b6377bb8db4f038c6ae369d9906e80, State: Initialized, Role: FOLLOWER
I20260925 19:07:46.951416 16628 consensus_queue.cc:260] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:07:46.951670 16632 tablet_bootstrap.cc:492] T f5c7140d4c4c4482a881068379ceb2cb P b8ebe057555540bf8d65ba18ee36a54f: No bootstrap required, opened a new log
I20260925 19:07:46.951719 16628 ts_tablet_manager.cc:1428] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 19:07:46.951781 16632 ts_tablet_manager.cc:1397] T f5c7140d4c4c4482a881068379ceb2cb P b8ebe057555540bf8d65ba18ee36a54f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 19:07:46.952140 16629 tablet_bootstrap.cc:492] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e: No bootstrap required, opened a new log
I20260925 19:07:46.952224 16629 ts_tablet_manager.cc:1397] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260925 19:07:46.953835 16632 raft_consensus.cc:348] T f5c7140d4c4c4482a881068379ceb2cb P b8ebe057555540bf8d65ba18ee36a54f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:07:46.954032 16632 raft_consensus.cc:374] T f5c7140d4c4c4482a881068379ceb2cb P b8ebe057555540bf8d65ba18ee36a54f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:07:46.954068 16632 raft_consensus.cc:729] T f5c7140d4c4c4482a881068379ceb2cb P b8ebe057555540bf8d65ba18ee36a54f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ebe057555540bf8d65ba18ee36a54f, State: Initialized, Role: FOLLOWER
I20260925 19:07:46.951797 16477 heartbeater.cc:503] Master 127.0.196.254:41039 was elected leader, sending a full tablet report...
I20260925 19:07:46.954193 16632 consensus_queue.cc:260] T f5c7140d4c4c4482a881068379ceb2cb P b8ebe057555540bf8d65ba18ee36a54f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:07:46.954496 16632 ts_tablet_manager.cc:1428] T f5c7140d4c4c4482a881068379ceb2cb P b8ebe057555540bf8d65ba18ee36a54f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:07:46.954481 16629 raft_consensus.cc:348] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:07:46.954712 16629 raft_consensus.cc:374] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:07:46.954726 16611 heartbeater.cc:503] Master 127.0.196.254:41039 was elected leader, sending a full tablet report...
I20260925 19:07:46.954758 16629 raft_consensus.cc:729] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32b8f17051084c27a6ef472c2fbb312e, State: Initialized, Role: FOLLOWER
I20260925 19:07:46.954867 16629 consensus_queue.cc:260] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:07:46.955144 16629 ts_tablet_manager.cc:1428] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:07:46.955277 16343 heartbeater.cc:503] Master 127.0.196.254:41039 was elected leader, sending a full tablet report...
W20260925 19:07:46.964082 16478 tablet.cc:2369] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:07:47.033238 16344 tablet.cc:2369] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:07:47.103261 16634 raft_consensus.cc:482] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:07:47.103425 16634 raft_consensus.cc:504] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:07:47.103891 16634 leader_election.cc:288] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8ebe057555540bf8d65ba18ee36a54f (127.0.196.195:43139), 32b8f17051084c27a6ef472c2fbb312e (127.0.196.193:35643)
I20260925 19:07:47.107086 16479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:47.104103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:35643 (local address 127.0.196.194:53951)
0925 19:07:47.104186 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:47.104200 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:47.104252 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:47.104442 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:47.104443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:47.104445 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:07:47.104527 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:47.104529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:47.105213 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:47.105215 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:47.106066 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:47.106068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:47.106789 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:47.106790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:47.106884 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:47.106889 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:07:47.106953 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 19:07:47.106996 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 19:07:47.107147 16626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:47.104188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:53951 (local address 127.0.196.193:35643)
0925 19:07:47.104237 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:47.104240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:47.104252 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:07:47.104329 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:47.104330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:47.104342 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:47.104432 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:07:47.104608 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:47.105132 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:47.106158 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:47.106720 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:47.106778 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:47.106785 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:47.107000 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 19:07:47.107040 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:07:47.107573 16637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:47.104103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:43139 (local address 127.0.196.194:59971)
0925 19:07:47.104325 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:47.104336 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:07:47.104416 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:07:47.104542 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:07:47.104543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:07:47.104544 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:07:47.104637 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:47.104639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:47.105435 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:47.105436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:47.106320 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:07:47.106323 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:07:47.107331 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:07:47.107332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:07:47.107405 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:47.107409 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:07:47.107460 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 19:07:47.107496 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":41,"thread_start_us":105,"threads_started":1}
I20260925 19:07:47.107592 16627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:47.104195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:59971 (local address 127.0.196.195:43139)
0925 19:07:47.104237 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:47.104240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:47.104424 (+ 184us) server_negotiation.cc:407] Connection header received
0925 19:07:47.104462 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:47.104463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:47.104473 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:47.104540 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:07:47.104713 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:47.105343 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:47.106456 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:47.107039 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:47.107280 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:47.107293 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:47.107485 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 19:07:47.107532 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:07:47.107520 16297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c7140d4c4c4482a881068379ceb2cb" candidate_uuid: "e8b6377bb8db4f038c6ae369d9906e80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32b8f17051084c27a6ef472c2fbb312e" is_pre_election: true
I20260925 19:07:47.107708 16297 raft_consensus.cc:2454] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8b6377bb8db4f038c6ae369d9906e80 in term 0.
I20260925 19:07:47.107887 16565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c7140d4c4c4482a881068379ceb2cb" candidate_uuid: "e8b6377bb8db4f038c6ae369d9906e80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8ebe057555540bf8d65ba18ee36a54f" is_pre_election: true
I20260925 19:07:47.107983 16366 leader_election.cc:302] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32b8f17051084c27a6ef472c2fbb312e, e8b6377bb8db4f038c6ae369d9906e80; no voters:
I20260925 19:07:47.108031 16565 raft_consensus.cc:2454] T f5c7140d4c4c4482a881068379ceb2cb P b8ebe057555540bf8d65ba18ee36a54f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8b6377bb8db4f038c6ae369d9906e80 in term 0.
I20260925 19:07:47.108158 16634 raft_consensus.cc:2781] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:07:47.108217 16634 raft_consensus.cc:482] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:07:47.108254 16634 raft_consensus.cc:3037] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:07:47.109020 16634 raft_consensus.cc:504] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:07:47.109185 16634 leader_election.cc:288] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80 [CANDIDATE]: Term 1 election: Requested vote from peers b8ebe057555540bf8d65ba18ee36a54f (127.0.196.195:43139), 32b8f17051084c27a6ef472c2fbb312e (127.0.196.193:35643)
I20260925 19:07:47.109401 16565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c7140d4c4c4482a881068379ceb2cb" candidate_uuid: "e8b6377bb8db4f038c6ae369d9906e80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8ebe057555540bf8d65ba18ee36a54f"
I20260925 19:07:47.109407 16297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c7140d4c4c4482a881068379ceb2cb" candidate_uuid: "e8b6377bb8db4f038c6ae369d9906e80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32b8f17051084c27a6ef472c2fbb312e"
I20260925 19:07:47.109495 16565 raft_consensus.cc:3037] T f5c7140d4c4c4482a881068379ceb2cb P b8ebe057555540bf8d65ba18ee36a54f [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:07:47.109496 16297 raft_consensus.cc:3037] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:07:47.110445 16565 raft_consensus.cc:2454] T f5c7140d4c4c4482a881068379ceb2cb P b8ebe057555540bf8d65ba18ee36a54f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8b6377bb8db4f038c6ae369d9906e80 in term 1.
I20260925 19:07:47.110445 16297 raft_consensus.cc:2454] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8b6377bb8db4f038c6ae369d9906e80 in term 1.
I20260925 19:07:47.110700 16367 leader_election.cc:302] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8ebe057555540bf8d65ba18ee36a54f, e8b6377bb8db4f038c6ae369d9906e80; no voters:
I20260925 19:07:47.111227 16634 raft_consensus.cc:2781] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:07:47.111408 16634 raft_consensus.cc:686] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80 [term 1 LEADER]: Becoming Leader. State: Replica: e8b6377bb8db4f038c6ae369d9906e80, State: Running, Role: LEADER
I20260925 19:07:47.111519 16634 consensus_queue.cc:237] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:07:47.112361 16152 catalog_manager.cc:5495] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8b6377bb8db4f038c6ae369d9906e80 (127.0.196.194). New cstate: current_term: 1 leader_uuid: "e8b6377bb8db4f038c6ae369d9906e80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } health_report { overall_health: UNKNOWN } } }
I20260925 19:07:47.121217 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:47.118640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52502 (local address 127.0.196.194:43989)
0925 19:07:47.118680 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 19:07:47.118683 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:07:47.118691 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:07:47.118773 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:07:47.118774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:07:47.118803 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:07:47.118884 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:07:47.119031 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:47.119559 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:47.120359 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:07:47.120560 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:07:47.120682 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:07:47.120860 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:07:47.120987 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:07:47.121086 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 19:07:47.121104 (+ 18us) server_negotiation.cc:299] Negotiation successful
0925 19:07:47.121126 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:07:47.123383 16297 raft_consensus.cc:1261] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e [term 1 FOLLOWER]: Refusing update from remote peer e8b6377bb8db4f038c6ae369d9906e80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:07:47.123437 16565 raft_consensus.cc:1261] T f5c7140d4c4c4482a881068379ceb2cb P b8ebe057555540bf8d65ba18ee36a54f [term 1 FOLLOWER]: Refusing update from remote peer e8b6377bb8db4f038c6ae369d9906e80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:07:47.123813 16638 consensus_queue.cc:1035] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:07:47.123970 16634 consensus_queue.cc:1035] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:07:47.126291 16641 mvcc.cc:205] Tried to move back new op lower bound from 7333327942133989376 to 7333327942089400320. Current Snapshot: MvccSnapshot[applied={T|T < 7333327942133989376}]
I20260925 19:07:47.126677 16643 mvcc.cc:205] Tried to move back new op lower bound from 7333327942133989376 to 7333327942089400320. Current Snapshot: MvccSnapshot[applied={T|T < 7333327942133989376}]
I20260925 19:07:47.128614 16152 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:07:47.128738 16152 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:07:47.129468 16152 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c: Sending DeleteTablet for 3 replicas of tablet f5c7140d4c4c4482a881068379ceb2cb
I20260925 19:07:47.129776 16411 tablet_service.cc:1514] Processing DeleteTablet for tablet f5c7140d4c4c4482a881068379ceb2cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:07:47 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:52494
I20260925 19:07:47.129804 16277 tablet_service.cc:1514] Processing DeleteTablet for tablet f5c7140d4c4c4482a881068379ceb2cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:07:47 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:43996
I20260925 19:07:47.130029 16545 tablet_service.cc:1514] Processing DeleteTablet for tablet f5c7140d4c4c4482a881068379ceb2cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:07:47 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:50436
I20260925 19:07:47.130152 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 16214
I20260925 19:07:47.130956 16653 tablet_replica.cc:330] T f5c7140d4c4c4482a881068379ceb2cb P b8ebe057555540bf8d65ba18ee36a54f: stopping tablet replica
I20260925 19:07:47.131104 16653 raft_consensus.cc:2229] T f5c7140d4c4c4482a881068379ceb2cb P b8ebe057555540bf8d65ba18ee36a54f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:07:47.131251 16653 raft_consensus.cc:2258] T f5c7140d4c4c4482a881068379ceb2cb P b8ebe057555540bf8d65ba18ee36a54f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:07:47.131263 16651 tablet_replica.cc:330] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80: stopping tablet replica
I20260925 19:07:47.131359 16651 raft_consensus.cc:2229] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:07:47.131589 16651 raft_consensus.cc:2258] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:07:47.131800 16653 ts_tablet_manager.cc:1905] T f5c7140d4c4c4482a881068379ceb2cb P b8ebe057555540bf8d65ba18ee36a54f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:07:47.132123 16651 ts_tablet_manager.cc:1905] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:07:47.133549 16653 ts_tablet_manager.cc:1918] T f5c7140d4c4c4482a881068379ceb2cb P b8ebe057555540bf8d65ba18ee36a54f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:07:47.133662 16653 log.cc:1196] T f5c7140d4c4c4482a881068379ceb2cb P b8ebe057555540bf8d65ba18ee36a54f: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/wal/wals/f5c7140d4c4c4482a881068379ceb2cb
I20260925 19:07:47.133872 16651 ts_tablet_manager.cc:1918] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:07:47.133955 16651 log.cc:1196] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/wal/wals/f5c7140d4c4c4482a881068379ceb2cb
I20260925 19:07:47.133982 16653 ts_tablet_manager.cc:1939] T f5c7140d4c4c4482a881068379ceb2cb P b8ebe057555540bf8d65ba18ee36a54f: Deleting consensus metadata
I20260925 19:07:47.134249 16651 ts_tablet_manager.cc:1939] T f5c7140d4c4c4482a881068379ceb2cb P e8b6377bb8db4f038c6ae369d9906e80: Deleting consensus metadata
I20260925 19:07:47.134639 16139 catalog_manager.cc:4841] TS b8ebe057555540bf8d65ba18ee36a54f (127.0.196.195:43139): tablet f5c7140d4c4c4482a881068379ceb2cb (table test-table [id=fea2be3a872542488958f2ad20ac82f8]) successfully deleted
I20260925 19:07:47.134668 16137 catalog_manager.cc:4841] TS e8b6377bb8db4f038c6ae369d9906e80 (127.0.196.194:43989): tablet f5c7140d4c4c4482a881068379ceb2cb (table test-table [id=fea2be3a872542488958f2ad20ac82f8]) successfully deleted
W20260925 19:07:47.142846 16138 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.196.193:35643 (error 108)
I20260925 19:07:47.143474 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:07:47.143325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:35643 (local address 127.0.0.1:44002)
0925 19:07:47.143386 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 19:07:47.143428 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.193:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 19:07:47.143522 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 16348
W20260925 19:07:47.145107 16138 catalog_manager.cc:4584] TS 32b8f17051084c27a6ef472c2fbb312e (127.0.196.193:35643): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f5c7140d4c4c4482a881068379ceb2cb: Network error: Client connection negotiation failed: client connection to 127.0.196.193:35643: connect: Connection refused (error 111)
I20260925 19:07:47.148612 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 16482
I20260925 19:07:47.154422 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 16120
I20260925 19:08:07.161027 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:41039
--webserver_interface=127.0.196.254
--webserver_port=35695
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:37989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:41039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:08:07.243676 16655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:07.243917 16655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:07.243942 16655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:07.245251 16655 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 19:08:07.245311 16655 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 19:08:07.245325 16655 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 19:08:07.245337 16655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:08:07.245353 16655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:07.245366 16655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:07.245379 16655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:08:07.245438 16655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:08:07.247196 16655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:37989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:41039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:41039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=35695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:07.247498 16655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:07.247822 16655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:08:07.257347 16655 server_base.cc:1034] running on GCE node
W20260925 19:08:07.257352 16661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:07.257349 16662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:07.257594 16664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:07.257882 16655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:07.258303 16655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:07.259498 16655 hybrid_clock.cc:648] HybridClock initialized: now 1790363287259464 us; error 57 us; skew 500 ppm
Sep 25 19:08:07 dist-test-slave-rn98 krb5kdc[16107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:08:07.268824 16655 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:08:07.269433 16655 webserver.cc:460] Webserver started at http://127.0.196.254:35695/ using document root <none> and password file <none>
I20260925 19:08:07.269711 16655 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:07.269811 16655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:07.271525 16655 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:08:07.272341 16671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:07.272558 16655 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:08:07.272665 16655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "283208106c934ffd993371388e6acb4c"
format_stamp: "Formatted at 2026-09-25 19:07:46 on dist-test-slave-rn98"
server_key: "05906fd3fcc23ff6e80894adc84c17eb"
server_key_iv: "7be0c67088632fc1166fc2f6c2bda3f6"
server_key_version: "encryptionkey@0"
I20260925 19:08:07.273141 16655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:07.312134 16655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:07.314911 16655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:07.315093 16655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:07.319372 16655 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:41039
I20260925 19:08:07.319460 16723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:41039 every 8 connection(s)
I20260925 19:08:07.319868 16655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:08:07.321691 16724 sys_catalog.cc:263] Verifying existing consensus state
I20260925 19:08:07.322407 16724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c: Bootstrap starting.
I20260925 19:08:07.327358 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 16655
I20260925 19:08:07.327742 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:35643
--local_ip_for_outbound_sockets=127.0.196.193
--tserver_master_addrs=127.0.196.254:41039
--webserver_port=34401
--webserver_interface=127.0.196.193
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:37989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:08:07.331377 16724 log.cc:824] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:07.333505 16724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:08:07.333887 16724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c: Bootstrap complete.
I20260925 19:08:07.336388 16724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283208106c934ffd993371388e6acb4c" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41039 } }
I20260925 19:08:07.336673 16724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283208106c934ffd993371388e6acb4c, State: Initialized, Role: FOLLOWER
I20260925 19:08:07.336905 16724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283208106c934ffd993371388e6acb4c" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41039 } }
I20260925 19:08:07.337005 16724 raft_consensus.cc:388] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:08:07.337069 16724 raft_consensus.cc:482] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:08:07.337118 16724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [term 1 FOLLOWER]: Advancing to term 2
I20260925 19:08:07.338223 16724 raft_consensus.cc:504] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283208106c934ffd993371388e6acb4c" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41039 } }
I20260925 19:08:07.338358 16724 leader_election.cc:302] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 283208106c934ffd993371388e6acb4c; no voters:
I20260925 19:08:07.338577 16724 leader_election.cc:288] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 19:08:07.338637 16728 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [term 2 FOLLOWER]: Leader election won for term 2
I20260925 19:08:07.338759 16728 raft_consensus.cc:686] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [term 2 LEADER]: Becoming Leader. State: Replica: 283208106c934ffd993371388e6acb4c, State: Running, Role: LEADER
I20260925 19:08:07.338908 16724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:08:07.338924 16728 consensus_queue.cc:237] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283208106c934ffd993371388e6acb4c" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41039 } }
I20260925 19:08:07.339353 16729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "283208106c934ffd993371388e6acb4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283208106c934ffd993371388e6acb4c" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41039 } } }
I20260925 19:08:07.339442 16729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [sys.catalog]: This master's current role is: LEADER
I20260925 19:08:07.339767 16730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 283208106c934ffd993371388e6acb4c. Latest consensus state: current_term: 2 leader_uuid: "283208106c934ffd993371388e6acb4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283208106c934ffd993371388e6acb4c" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41039 } } }
I20260925 19:08:07.339835 16734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:08:07.339916 16730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [sys.catalog]: This master's current role is: LEADER
I20260925 19:08:07.340794 16734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:08:07.341008 16734 catalog_manager.cc:1259] Loaded cluster ID: 3a354f1db28945af8e5b6ed8ba2b7cb0
I20260925 19:08:07.341152 16734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:08:07.342453 16734 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:08:07.342681 16734 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c: Loaded TSK: 0
I20260925 19:08:07.357630 16734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c: Generated new TSK 1
I20260925 19:08:07.357775 16734 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 19:08:07.419667 16726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:07.419930 16726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:07.419974 16726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:07.421415 16726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:07.421495 16726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:07.421554 16726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:08:07.423238 16726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:37989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:35643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=34401
--tserver_master_addrs=127.0.196.254:41039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:07.423538 16726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:07.423802 16726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:07.429947 16751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:07.430012 16752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:07.430034 16754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:07.430510 16726 server_base.cc:1034] running on GCE node
I20260925 19:08:07.430786 16726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:07.431100 16726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:07.432264 16726 hybrid_clock.cc:648] HybridClock initialized: now 1790363287432247 us; error 34 us; skew 500 ppm
Sep 25 19:08:07 dist-test-slave-rn98 krb5kdc[16107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:08:07.435276 16726 init.cc:377] Logged in from keytab as kudu/127.0.196.193@KRBTEST.COM (short username kudu)
I20260925 19:08:07.435881 16726 webserver.cc:460] Webserver started at http://127.0.196.193:34401/ using document root <none> and password file <none>
I20260925 19:08:07.436167 16726 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:07.436249 16726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:07.438187 16726 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:08:07.439059 16761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:07.439373 16726 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:08:07.439479 16726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "32b8f17051084c27a6ef472c2fbb312e"
format_stamp: "Formatted at 2026-09-25 19:07:46 on dist-test-slave-rn98"
server_key: "95bb34c2589216a0420e9cdce5fb14cd"
server_key_iv: "9e4d076f09bf6a566d12e28957e4dd2f"
server_key_version: "encryptionkey@0"
I20260925 19:08:07.439967 16726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:07.458035 16726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:07.460928 16726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:07.461117 16726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:07.461395 16726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:08:07.461992 16768 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 19:08:07.463706 16726 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 19:08:07.463771 16726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 19:08:07.463804 16726 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 19:08:07.464637 16726 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 19:08:07.464679 16726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 19:08:07.464773 16768 tablet_bootstrap.cc:492] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e: Bootstrap starting.
I20260925 19:08:07.471657 16726 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:35643
I20260925 19:08:07.471884 16875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:35643 every 8 connection(s)
I20260925 19:08:07.472286 16726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
I20260925 19:08:07.473456 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 16726
I20260925 19:08:07.473801 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:43989
--local_ip_for_outbound_sockets=127.0.196.194
--tserver_master_addrs=127.0.196.254:41039
--webserver_port=37129
--webserver_interface=127.0.196.194
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:37989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:08:07.476325 16768 log.cc:824] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:07.479298 16768 tablet_bootstrap.cc:492] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:08:07.479628 16768 tablet_bootstrap.cc:492] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e: Bootstrap complete.
I20260925 19:08:07.480850 16768 ts_tablet_manager.cc:1397] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260925 19:08:07.482836 16768 raft_consensus.cc:348] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:08:07.483309 16768 raft_consensus.cc:729] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32b8f17051084c27a6ef472c2fbb312e, State: Initialized, Role: FOLLOWER
I20260925 19:08:07.483512 16768 consensus_queue.cc:260] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:08:07.483848 16768 ts_tablet_manager.cc:1428] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
Sep 25 19:08:07 dist-test-slave-rn98 krb5kdc[16107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:08:07.494462 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:07.481683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.193:42877)
0925 19:08:07.482370 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:07.482427 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:07.482833 (+ 406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:07.483535 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:07.483540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:07.483995 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:08:07.484341 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:07.484348 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.485925 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.485927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:07.487281 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:07.487284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.488168 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.488169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:07.488315 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:07.488940 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:08:07.488951 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:08:07.490568 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:08:07.492679 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:07.492688 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:07.492690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:07.492942 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:07.493233 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:07.493234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:07.493235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:07.493358 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:07.493774 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:08:07.493782 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:08:07.493930 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 19:08:07.494036 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 19:08:07.494246 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":103,"threads_started":1}
I20260925 19:08:07.494460 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:07.481745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:42877 (local address 127.0.196.254:41039)
0925 19:08:07.482225 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:07.482237 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:07.482693 (+ 456us) server_negotiation.cc:407] Connection header received
0925 19:08:07.483151 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:07.483159 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:07.483227 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:07.483370 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:07.484510 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.485786 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.487397 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.488011 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.488144 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:07.490809 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:07.490814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:07.490815 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:08:07.490826 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:07.492562 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:07.493042 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:07.493048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:07.493049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:07.493159 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:07.493436 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:07.493437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:07.493437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:07.493691 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:07.493784 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:07.494119 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 19:08:07.494258 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":98,"threads_started":1}
I20260925 19:08:07.495368 16876 heartbeater.cc:347] Connected to a master server at 127.0.196.254:41039
I20260925 19:08:07.495486 16876 heartbeater.cc:464] Registering TS with master...
I20260925 19:08:07.495671 16876 heartbeater.cc:511] Master 127.0.196.254:41039 requested a full tablet report, sending...
I20260925 19:08:07.496479 16688 ts_manager.cc:194] Registered new tserver with Master: 32b8f17051084c27a6ef472c2fbb312e (127.0.196.193:35643)
I20260925 19:08:07.497685 16688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.193@KRBTEST.COM'} at 127.0.196.193:42877
I20260925 19:08:07.498801 16876 heartbeater.cc:503] Master 127.0.196.254:41039 was elected leader, sending a full tablet report...
Sep 25 19:08:07 dist-test-slave-rn98 krb5kdc[16107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for kudu/127.0.196.193@KRBTEST.COM
I20260925 19:08:07.506480 16883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:07.497152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:35643 (local address 127.0.0.1:54758)
0925 19:08:07.497342 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:07.497378 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:07.497442 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:07.498321 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:07.498327 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:07.498608 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:08:07.498728 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:07.498731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.499790 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.499791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:07.500392 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:07.500394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.500892 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.500894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:07.500980 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:07.501522 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:08:07.501526 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:08:07.503183 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:08:07.505134 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:07.505144 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:07.505146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:07.505384 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:07.505628 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:07.505629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:07.505629 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:07.505701 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:07.506048 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:08:07.506056 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:08:07.506118 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 19:08:07.506220 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 19:08:07.506369 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":122,"threads_started":1}
I20260925 19:08:07.506480 16884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:07.497987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54758 (local address 127.0.196.193:35643)
0925 19:08:07.498174 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:07.498179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:07.498195 (+ 16us) server_negotiation.cc:407] Connection header received
0925 19:08:07.498223 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:07.498224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:07.498236 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:07.498321 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:07.498815 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.499705 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.500488 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.500796 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.500889 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:07.503295 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:07.503307 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:07.503307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:08:07.503317 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:07.505025 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:07.505481 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:07.505484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:07.505485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:07.505549 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:07.505801 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:07.505802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:07.505803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:07.505967 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:07.506057 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:07.506255 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 19:08:07.506368 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":100,"threads_started":1}
I20260925 19:08:07.507221 16810 tablet_service.cc:1514] Processing DeleteTablet for tablet f5c7140d4c4c4482a881068379ceb2cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:07:47 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:54758
I20260925 19:08:07.507937 16885 tablet_replica.cc:330] stopping tablet replica
I20260925 19:08:07.508069 16885 raft_consensus.cc:2229] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:08:07.508167 16885 raft_consensus.cc:2258] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:08:07.508526 16885 ts_tablet_manager.cc:1905] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:08:07.510605 16885 ts_tablet_manager.cc:1918] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:08:07.510771 16885 log.cc:1196] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/wal/wals/f5c7140d4c4c4482a881068379ceb2cb
I20260925 19:08:07.511026 16885 ts_tablet_manager.cc:1939] T f5c7140d4c4c4482a881068379ceb2cb P 32b8f17051084c27a6ef472c2fbb312e: Deleting consensus metadata
I20260925 19:08:07.511587 16674 catalog_manager.cc:4841] TS 32b8f17051084c27a6ef472c2fbb312e (127.0.196.193:35643): tablet f5c7140d4c4c4482a881068379ceb2cb (table test-table [id=fea2be3a872542488958f2ad20ac82f8]) successfully deleted
W20260925 19:08:07.565008 16878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:07.565253 16878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:07.565279 16878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:07.566463 16878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:07.566504 16878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:07.566533 16878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:08:07.568148 16878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:37989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:43989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=37129
--tserver_master_addrs=127.0.196.254:41039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:07.568369 16878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:07.568568 16878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:07.574383 16893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:07.574370 16890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:07.574493 16878 server_base.cc:1034] running on GCE node
W20260925 19:08:07.574370 16891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:07.574937 16878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:07.575268 16878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:07.576509 16878 hybrid_clock.cc:648] HybridClock initialized: now 1790363287576486 us; error 49 us; skew 500 ppm
Sep 25 19:08:07 dist-test-slave-rn98 krb5kdc[16107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:08:07.583321 16878 init.cc:377] Logged in from keytab as kudu/127.0.196.194@KRBTEST.COM (short username kudu)
I20260925 19:08:07.583886 16878 webserver.cc:460] Webserver started at http://127.0.196.194:37129/ using document root <none> and password file <none>
I20260925 19:08:07.584133 16878 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:07.584219 16878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:07.585731 16878 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:08:07.586445 16900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:07.586707 16878 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:08:07.586814 16878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "e8b6377bb8db4f038c6ae369d9906e80"
format_stamp: "Formatted at 2026-09-25 19:07:46 on dist-test-slave-rn98"
server_key: "5ac6036f557c209eee2b864d9e8c137e"
server_key_iv: "be22d3fe09633d6c66857d29c19f172f"
server_key_version: "encryptionkey@0"
I20260925 19:08:07.587216 16878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:07.597702 16878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:07.600551 16878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:07.600982 16878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:07.601421 16878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:08:07.601831 16878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:08:07.601904 16878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:07.601951 16878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:08:07.601981 16878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:07.608687 16878 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:43989
I20260925 19:08:07.608731 17013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:43989 every 8 connection(s)
I20260925 19:08:07.609128 16878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
I20260925 19:08:07.610455 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 16878
I20260925 19:08:07.610795 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:43139
--local_ip_for_outbound_sockets=127.0.196.195
--tserver_master_addrs=127.0.196.254:41039
--webserver_port=40171
--webserver_interface=127.0.196.195
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:37989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 19:08:07 dist-test-slave-rn98 krb5kdc[16107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:08:07.620759 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:07.610023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:33159 (local address 127.0.196.254:41039)
0925 19:08:07.610118 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:07.610121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:07.610425 (+ 304us) server_negotiation.cc:407] Connection header received
0925 19:08:07.611254 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:07.611255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:07.611271 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:07.611345 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:07.612684 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.613200 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.614340 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.614780 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.614879 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:07.617147 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:07.617150 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:07.617150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:08:07.617161 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:07.619011 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:07.619546 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:07.619547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:07.619548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:07.619619 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:07.619886 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:07.619887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:07.619887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:07.620060 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:07.620165 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:07.620467 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 19:08:07.620613 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 19:08:07.620803 17016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:07.609975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.194:33159)
0925 19:08:07.610258 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:07.610312 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:07.611110 (+ 798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:07.611423 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:07.611428 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:07.612302 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:08:07.612560 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:07.612567 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.613312 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.613314 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:07.614163 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:07.614165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.614883 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.614884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:07.615002 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:07.615577 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:08:07.615586 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:08:07.617015 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:08:07.619108 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:07.619117 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:07.619119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:07.619468 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:07.619697 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:07.619698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:07.619698 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:07.619813 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:07.620150 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:08:07.620155 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:08:07.620297 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 19:08:07.620424 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 19:08:07.620614 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":118,"threads_started":1}
I20260925 19:08:07.621698 17014 heartbeater.cc:347] Connected to a master server at 127.0.196.254:41039
I20260925 19:08:07.621831 17014 heartbeater.cc:464] Registering TS with master...
I20260925 19:08:07.622030 17014 heartbeater.cc:511] Master 127.0.196.254:41039 requested a full tablet report, sending...
I20260925 19:08:07.622448 16688 ts_manager.cc:194] Registered new tserver with Master: e8b6377bb8db4f038c6ae369d9906e80 (127.0.196.194:43989)
I20260925 19:08:07.622892 16688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.194@KRBTEST.COM'} at 127.0.196.194:33159
W20260925 19:08:07.702895 17017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:07.703372 17017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:07.703438 17017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:07.705900 17017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:07.705983 17017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:07.706053 17017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:08:07.709209 17017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:37989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:43139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=40171
--tserver_master_addrs=127.0.196.254:41039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:07.709568 17017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:07.709903 17017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:07.716842 17025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:07.716948 17023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:07.716836 17022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:07.717636 17017 server_base.cc:1034] running on GCE node
I20260925 19:08:07.717931 17017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:07.718286 17017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:07.719475 17017 hybrid_clock.cc:648] HybridClock initialized: now 1790363287719451 us; error 37 us; skew 500 ppm
Sep 25 19:08:07 dist-test-slave-rn98 krb5kdc[16107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:08:07.723196 17017 init.cc:377] Logged in from keytab as kudu/127.0.196.195@KRBTEST.COM (short username kudu)
I20260925 19:08:07.723817 17017 webserver.cc:460] Webserver started at http://127.0.196.195:40171/ using document root <none> and password file <none>
I20260925 19:08:07.724145 17017 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:07.724244 17017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:07.726228 17017 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:08:07.726970 17032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:07.727180 17017 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:08:07.727284 17017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "b8ebe057555540bf8d65ba18ee36a54f"
format_stamp: "Formatted at 2026-09-25 19:07:46 on dist-test-slave-rn98"
server_key: "f5c943bf10f5c61d4cbc853dc01c4209"
server_key_iv: "bf0013e28242eb59c07c7651e107d25d"
server_key_version: "encryptionkey@0"
I20260925 19:08:07.727888 17017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:07.767911 17017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:07.770177 17017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:07.770336 17017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:07.770684 17017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:08:07.771063 17017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:08:07.771098 17017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:07.771122 17017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:08:07.771135 17017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:07.777352 17017 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:43139
I20260925 19:08:07.777392 17145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:43139 every 8 connection(s)
I20260925 19:08:07.777796 17017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
Sep 25 19:08:07 dist-test-slave-rn98 krb5kdc[16107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:08:07.787309 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 17017
I20260925 19:08:07.787935 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:07.778692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:59163 (local address 127.0.196.254:41039)
0925 19:08:07.778754 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:07.778758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:07.779030 (+ 272us) server_negotiation.cc:407] Connection header received
0925 19:08:07.779533 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:07.779535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:07.779549 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:07.779616 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:07.780632 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.781132 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.782165 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.782368 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.782427 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:07.784874 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:07.784875 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:07.784876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:08:07.784885 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:07.786306 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:07.786809 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:07.786810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:07.786811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:07.786868 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:07.787134 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:07.787135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:07.787135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:07.787307 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:07.787379 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:07.787746 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 19:08:07.787844 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:08:07.788141 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:07.778678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.195:59163)
0925 19:08:07.778866 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:07.778918 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:07.779435 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:07.779698 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:07.779702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:07.780309 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:08:07.780514 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:07.780519 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.781225 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.781226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:07.782078 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:07.782081 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.782461 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.782462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:07.782577 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:07.783244 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:08:07.783253 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:08:07.784738 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:08:07.786393 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:07.786400 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:07.786403 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:07.786735 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:07.786934 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:07.786934 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:07.786934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:07.787044 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:07.787401 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:08:07.787410 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:08:07.787566 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 19:08:07.787681 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 19:08:07.787865 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
I20260925 19:08:07.788805 17146 heartbeater.cc:347] Connected to a master server at 127.0.196.254:41039
I20260925 19:08:07.788897 17146 heartbeater.cc:464] Registering TS with master...
I20260925 19:08:07.789115 17146 heartbeater.cc:511] Master 127.0.196.254:41039 requested a full tablet report, sending...
I20260925 19:08:07.789556 16688 ts_manager.cc:194] Registered new tserver with Master: b8ebe057555540bf8d65ba18ee36a54f (127.0.196.195:43139)
I20260925 19:08:07.790093 16688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.195@KRBTEST.COM'} at 127.0.196.195:59163
I20260925 19:08:07.794526 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:07.787728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45684 (local address 127.0.196.254:41039)
0925 19:08:07.788061 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:07.788063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:07.788070 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:08:07.788089 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:07.788090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:07.788100 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:07.788162 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:07.789013 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.789648 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.790725 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.790941 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.790998 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:07.792309 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:07.792310 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:07.792310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:08:07.792316 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:07.793555 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:07.793895 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:07.793896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:07.793896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:07.793944 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:07.794109 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:07.794109 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:07.794110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:07.794207 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:07.794338 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:07.794365 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 19:08:07.794422 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":70,"threads_started":1}
I20260925 19:08:07.795346 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:08:07.798053 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:07.795708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45690 (local address 127.0.196.254:41039)
0925 19:08:07.795745 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:07.795747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:07.795823 (+ 76us) server_negotiation.cc:407] Connection header received
0925 19:08:07.795882 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:07.795883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:07.795913 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:07.795974 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:08:07.796089 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.796545 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.797369 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.797550 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.797609 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:07.797828 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:08:07.797876 (+ 48us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 19:08:07.797999 (+ 123us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45690: token expired
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 19:08:07.798100 16881 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45690: token expired
W20260925 19:08:07.798276 16615 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.196.254:41039)
I20260925 19:08:07.804742 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:07.798577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45700 (local address 127.0.196.254:41039)
0925 19:08:07.798700 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:07.798702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:07.798752 (+ 50us) server_negotiation.cc:407] Connection header received
0925 19:08:07.798769 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:07.798770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:07.798777 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:07.798822 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:07.799285 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.799736 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.800418 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.800585 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.800633 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:07.801996 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:07.801997 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:07.801997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:08:07.802004 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:07.803638 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:07.804031 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:07.804032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:07.804032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:07.804088 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:07.804266 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:07.804267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:07.804267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:07.804374 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:07.804448 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:07.804585 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 19:08:07.804654 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 19:08:07.806041 16688 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:08:07.807828 16688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:08:07.810063 16810 tablet_service.cc:1467] Processing CreateTablet for tablet 4082737c22b24a4aac98f891ebc5fb7c (DEFAULT_TABLE table=test-table [id=62fdec5eb7984054bfac952f07eaf930]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:08:07.810281 16810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4082737c22b24a4aac98f891ebc5fb7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:07.811841 16768 tablet_bootstrap.cc:492] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e: Bootstrap starting.
I20260925 19:08:07.812206 16768 tablet_bootstrap.cc:654] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:07.812839 16768 tablet_bootstrap.cc:492] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e: No bootstrap required, opened a new log
I20260925 19:08:07.812893 16768 ts_tablet_manager.cc:1397] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 19:08:07.813284 16768 raft_consensus.cc:348] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:08:07.813371 16768 raft_consensus.cc:374] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:07.813421 16768 raft_consensus.cc:729] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32b8f17051084c27a6ef472c2fbb312e, State: Initialized, Role: FOLLOWER
I20260925 19:08:07.813472 16768 consensus_queue.cc:260] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:08:07.813580 16768 ts_tablet_manager.cc:1428] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:08:07.814734 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:07.810036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49876 (local address 127.0.196.194:43989)
0925 19:08:07.810240 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:07.810246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:07.810259 (+ 13us) server_negotiation.cc:407] Connection header received
0925 19:08:07.810308 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:07.810310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:07.810329 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:07.810610 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:07.810738 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.811916 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.813553 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.814325 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.814422 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:07.814479 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:07.814576 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 19:08:07.814636 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":115,"threads_started":1}
I20260925 19:08:07.814723 16883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:07.809746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:43989 (local address 127.0.0.1:49876)
0925 19:08:07.809790 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:07.809802 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:07.809845 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:07.810459 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:07.810460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:07.810462 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:07.810525 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:07.810527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.812021 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.812022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:07.813444 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:07.813448 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.814424 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.814425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:07.814500 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:07.814504 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:08:07.814561 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 19:08:07.814637 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":98}
I20260925 19:08:07.814731 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:07.810033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50238 (local address 127.0.196.195:43139)
0925 19:08:07.810295 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:07.810299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:07.810309 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:08:07.810343 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:07.810345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:07.810363 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:07.810442 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:07.810658 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.811930 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.813542 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.814310 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.814395 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:07.814460 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:07.814559 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 19:08:07.814599 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":121,"threads_started":1}
I20260925 19:08:07.814726 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:07.809746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:43139 (local address 127.0.0.1:50238)
0925 19:08:07.809883 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:07.809894 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:07.809946 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:07.810455 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:07.810457 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:07.810459 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:07.810531 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:07.810533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.812021 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.812022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:07.813442 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:07.813446 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.814394 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.814396 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:07.814460 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:07.814483 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 19:08:07.814543 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 19:08:07.814592 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":141,"thread_start_us":60,"threads_started":1}
I20260925 19:08:07.815459 17080 tablet_service.cc:1467] Processing CreateTablet for tablet 4082737c22b24a4aac98f891ebc5fb7c (DEFAULT_TABLE table=test-table [id=62fdec5eb7984054bfac952f07eaf930]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:08:07.815459 16948 tablet_service.cc:1467] Processing CreateTablet for tablet 4082737c22b24a4aac98f891ebc5fb7c (DEFAULT_TABLE table=test-table [id=62fdec5eb7984054bfac952f07eaf930]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:08:07.815802 17080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4082737c22b24a4aac98f891ebc5fb7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:07.815807 16948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4082737c22b24a4aac98f891ebc5fb7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:07.818821 17156 tablet_bootstrap.cc:492] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f: Bootstrap starting.
I20260925 19:08:07.819145 17157 tablet_bootstrap.cc:492] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80: Bootstrap starting.
I20260925 19:08:07.819839 17157 tablet_bootstrap.cc:654] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:07.819844 17156 tablet_bootstrap.cc:654] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:07.820492 17157 log.cc:824] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:07.820497 17156 log.cc:824] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:07.821434 17156 tablet_bootstrap.cc:492] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f: No bootstrap required, opened a new log
I20260925 19:08:07.821516 17156 ts_tablet_manager.cc:1397] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 19:08:07.821561 17157 tablet_bootstrap.cc:492] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80: No bootstrap required, opened a new log
I20260925 19:08:07.821632 17157 ts_tablet_manager.cc:1397] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 19:08:07.823647 17157 raft_consensus.cc:348] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:08:07.823647 17156 raft_consensus.cc:348] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:08:07.823866 17156 raft_consensus.cc:374] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:07.823866 17157 raft_consensus.cc:374] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:07.823905 17156 raft_consensus.cc:729] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ebe057555540bf8d65ba18ee36a54f, State: Initialized, Role: FOLLOWER
I20260925 19:08:07.823905 17157 raft_consensus.cc:729] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8b6377bb8db4f038c6ae369d9906e80, State: Initialized, Role: FOLLOWER
I20260925 19:08:07.824039 17157 consensus_queue.cc:260] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:08:07.824039 17156 consensus_queue.cc:260] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:08:07.824376 17157 ts_tablet_manager.cc:1428] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:08:07.824376 17156 ts_tablet_manager.cc:1428] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:08:07.824409 17146 heartbeater.cc:503] Master 127.0.196.254:41039 was elected leader, sending a full tablet report...
I20260925 19:08:07.824520 17014 heartbeater.cc:503] Master 127.0.196.254:41039 was elected leader, sending a full tablet report...
W20260925 19:08:07.859753 17015 tablet.cc:2369] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:08:07.862566 17161 raft_consensus.cc:482] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:08:07.862818 17161 raft_consensus.cc:504] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:08:07.863207 17161 leader_election.cc:288] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8b6377bb8db4f038c6ae369d9906e80 (127.0.196.194:43989), 32b8f17051084c27a6ef472c2fbb312e (127.0.196.193:35643)
I20260925 19:08:07.866489 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:07.863537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:36053 (local address 127.0.196.194:43989)
0925 19:08:07.863587 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:07.863590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:07.863868 (+ 278us) server_negotiation.cc:407] Connection header received
0925 19:08:07.863901 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:07.863902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:07.863913 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:07.863992 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:07.864221 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.864728 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.865706 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.866089 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.866148 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:07.866156 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:07.866335 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 19:08:07.866404 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:08:07.866501 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:07.863448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:35643 (local address 127.0.196.195:53427)
0925 19:08:07.863501 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:07.863593 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:07.863660 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:07.863889 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:07.863890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:07.863892 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:07.863976 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:07.863978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.864732 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.864733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:07.865508 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:07.865510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.866220 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.866222 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:07.866291 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:07.866314 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 19:08:07.866378 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 19:08:07.866421 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 19:08:07.866519 16884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:07.863590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:53427 (local address 127.0.196.193:35643)
0925 19:08:07.863637 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:07.863638 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:07.863650 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:08:07.863746 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:07.863747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:07.863777 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:07.863851 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:07.864088 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.864653 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.865648 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.866103 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.866202 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:07.866303 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:07.866399 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 19:08:07.866448 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 19:08:07.866503 17162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:07.863454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:43989 (local address 127.0.196.195:36053)
0925 19:08:07.863727 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:07.863738 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:07.863821 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:07.864056 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:07.864057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:07.864060 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:07.864139 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:07.864141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.864809 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.864810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:07.865547 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:07.865548 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.866174 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.866175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:07.866254 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:07.866257 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:08:07.866316 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 19:08:07.866362 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":64,"thread_start_us":125,"threads_started":1}
I20260925 19:08:07.866879 16830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4082737c22b24a4aac98f891ebc5fb7c" candidate_uuid: "b8ebe057555540bf8d65ba18ee36a54f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32b8f17051084c27a6ef472c2fbb312e" is_pre_election: true
I20260925 19:08:07.867077 16830 raft_consensus.cc:2454] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8ebe057555540bf8d65ba18ee36a54f in term 0.
I20260925 19:08:07.867204 16968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4082737c22b24a4aac98f891ebc5fb7c" candidate_uuid: "b8ebe057555540bf8d65ba18ee36a54f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8b6377bb8db4f038c6ae369d9906e80" is_pre_election: true
I20260925 19:08:07.867334 17035 leader_election.cc:302] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32b8f17051084c27a6ef472c2fbb312e, b8ebe057555540bf8d65ba18ee36a54f; no voters:
I20260925 19:08:07.867372 16968 raft_consensus.cc:2454] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8ebe057555540bf8d65ba18ee36a54f in term 0.
I20260925 19:08:07.867483 17161 raft_consensus.cc:2781] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:08:07.867545 17161 raft_consensus.cc:482] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:08:07.867571 17161 raft_consensus.cc:3037] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:07.868338 17161 raft_consensus.cc:504] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:08:07.868503 17161 leader_election.cc:288] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f [CANDIDATE]: Term 1 election: Requested vote from peers e8b6377bb8db4f038c6ae369d9906e80 (127.0.196.194:43989), 32b8f17051084c27a6ef472c2fbb312e (127.0.196.193:35643)
I20260925 19:08:07.868662 16830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4082737c22b24a4aac98f891ebc5fb7c" candidate_uuid: "b8ebe057555540bf8d65ba18ee36a54f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32b8f17051084c27a6ef472c2fbb312e"
I20260925 19:08:07.868736 16968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4082737c22b24a4aac98f891ebc5fb7c" candidate_uuid: "b8ebe057555540bf8d65ba18ee36a54f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8b6377bb8db4f038c6ae369d9906e80"
I20260925 19:08:07.868750 16830 raft_consensus.cc:3037] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:07.868808 16968 raft_consensus.cc:3037] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:07.869676 16830 raft_consensus.cc:2454] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8ebe057555540bf8d65ba18ee36a54f in term 1.
I20260925 19:08:07.869676 16968 raft_consensus.cc:2454] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8ebe057555540bf8d65ba18ee36a54f in term 1.
I20260925 19:08:07.869910 17035 leader_election.cc:302] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32b8f17051084c27a6ef472c2fbb312e, b8ebe057555540bf8d65ba18ee36a54f; no voters:
I20260925 19:08:07.870036 17161 raft_consensus.cc:2781] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:08:07.870224 17161 raft_consensus.cc:686] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f [term 1 LEADER]: Becoming Leader. State: Replica: b8ebe057555540bf8d65ba18ee36a54f, State: Running, Role: LEADER
I20260925 19:08:07.870333 17161 consensus_queue.cc:237] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:08:07.871402 16688 catalog_manager.cc:5495] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f reported cstate change: term changed from 0 to 1, leader changed from <none> to b8ebe057555540bf8d65ba18ee36a54f (127.0.196.195). New cstate: current_term: 1 leader_uuid: "b8ebe057555540bf8d65ba18ee36a54f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } health_report { overall_health: UNKNOWN } } }
I20260925 19:08:07.890450 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:07.887846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50242 (local address 127.0.196.195:43139)
0925 19:08:07.887916 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:07.887918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:07.887928 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:08:07.887977 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:07.887978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:07.887996 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:07.888078 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:08:07.888233 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.888711 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.889469 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:07.889710 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:07.889823 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:07.890000 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:08:07.890168 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:08:07.890280 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:07.890300 (+ 20us) server_negotiation.cc:299] Negotiation successful
0925 19:08:07.890325 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 19:08:07.892552 16830 raft_consensus.cc:1261] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e [term 1 FOLLOWER]: Refusing update from remote peer b8ebe057555540bf8d65ba18ee36a54f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:08:07.892601 16968 raft_consensus.cc:1261] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80 [term 1 FOLLOWER]: Refusing update from remote peer b8ebe057555540bf8d65ba18ee36a54f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:08:07.892944 17161 consensus_queue.cc:1035] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:08:07.893110 17163 consensus_queue.cc:1035] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f [LEADER]: Connected to new peer: Peer: permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:08:07.898192 16688 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:08:07.898324 16688 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:08:07.899170 16688 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c: Sending DeleteTablet for 3 replicas of tablet 4082737c22b24a4aac98f891ebc5fb7c
I20260925 19:08:07.899561 16948 tablet_service.cc:1514] Processing DeleteTablet for tablet 4082737c22b24a4aac98f891ebc5fb7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:08:07 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:49876
I20260925 19:08:07.899629 17080 tablet_service.cc:1514] Processing DeleteTablet for tablet 4082737c22b24a4aac98f891ebc5fb7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:08:07 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:50238
I20260925 19:08:07.899642 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 16726
I20260925 19:08:07.899787 16810 tablet_service.cc:1514] Processing DeleteTablet for tablet 4082737c22b24a4aac98f891ebc5fb7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:08:07 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:54758
I20260925 19:08:07.900434 17177 tablet_replica.cc:330] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f: stopping tablet replica
I20260925 19:08:07.900602 17177 raft_consensus.cc:2229] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:08:07.900717 17177 raft_consensus.cc:2258] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:08:07.901247 17177 ts_tablet_manager.cc:1905] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:08:07.903342 17177 ts_tablet_manager.cc:1918] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:08:07.903436 17177 log.cc:1196] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/wal/wals/4082737c22b24a4aac98f891ebc5fb7c
I20260925 19:08:07.903764 17177 ts_tablet_manager.cc:1939] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f: Deleting consensus metadata
I20260925 19:08:07.904348 16675 catalog_manager.cc:4841] TS b8ebe057555540bf8d65ba18ee36a54f (127.0.196.195:43139): tablet 4082737c22b24a4aac98f891ebc5fb7c (table test-table [id=62fdec5eb7984054bfac952f07eaf930]) successfully deleted
I20260925 19:08:07.906855 17176 tablet_replica.cc:330] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80: stopping tablet replica
I20260925 19:08:07.907042 17176 raft_consensus.cc:2229] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:08:07.907202 17176 raft_consensus.cc:2258] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:08:07.907800 17176 ts_tablet_manager.cc:1905] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 19:08:07.907804 16674 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.196.193:35643 (error 108)
I20260925 19:08:07.908213 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 16878
I20260925 19:08:07.908851 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:07.908309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:35643 (local address 127.0.0.1:54766)
0925 19:08:07.908760 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:07.908811 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.193:35643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427}
W20260925 19:08:07.909354 16674 catalog_manager.cc:4584] TS 32b8f17051084c27a6ef472c2fbb312e (127.0.196.193:35643): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4082737c22b24a4aac98f891ebc5fb7c: Network error: Client connection negotiation failed: client connection to 127.0.196.193:35643: connect: Connection refused (error 111)
I20260925 19:08:07.915887 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 17017
I20260925 19:08:07.917520 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:07.915967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:43989 (local address 127.0.0.1:49878)
0925 19:08:07.917418 (+ 1451us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:07.917466 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.194:43989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1419}
I20260925 19:08:07.921732 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 16655
I20260925 19:08:27.927660 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:41039
--webserver_interface=127.0.196.254
--webserver_port=35695
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:37989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:41039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:08:28.025727 17179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:28.025995 17179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:28.026022 17179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:28.027375 17179 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 19:08:28.027429 17179 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 19:08:28.027446 17179 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 19:08:28.027458 17179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:08:28.027474 17179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:28.027488 17179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:28.027503 17179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:08:28.027516 17179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:08:28.029273 17179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:37989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:41039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:41039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=35695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:28.029512 17179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:28.029718 17179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:28.035761 17184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:28.035764 17187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:28.035761 17185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:28.035786 17179 server_base.cc:1034] running on GCE node
I20260925 19:08:28.036273 17179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:28.036633 17179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:28.037791 17179 hybrid_clock.cc:648] HybridClock initialized: now 1790363308037766 us; error 37 us; skew 500 ppm
Sep 25 19:08:28 dist-test-slave-rn98 krb5kdc[16107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:08:28.040545 17179 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:08:28.041096 17179 webserver.cc:460] Webserver started at http://127.0.196.254:35695/ using document root <none> and password file <none>
I20260925 19:08:28.041378 17179 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:28.041452 17179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:28.042966 17179 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:08:28.043761 17194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:28.043936 17179 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:08:28.044051 17179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "283208106c934ffd993371388e6acb4c"
format_stamp: "Formatted at 2026-09-25 19:07:46 on dist-test-slave-rn98"
server_key: "05906fd3fcc23ff6e80894adc84c17eb"
server_key_iv: "7be0c67088632fc1166fc2f6c2bda3f6"
server_key_version: "encryptionkey@0"
I20260925 19:08:28.044488 17179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:28.055636 17179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:28.058100 17179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:28.058269 17179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:28.062450 17179 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:41039
I20260925 19:08:28.062487 17246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:41039 every 8 connection(s)
I20260925 19:08:28.062935 17179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:08:28.063640 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 17179
I20260925 19:08:28.063987 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:35643
--local_ip_for_outbound_sockets=127.0.196.193
--tserver_master_addrs=127.0.196.254:41039
--webserver_port=34401
--webserver_interface=127.0.196.193
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:37989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:08:28.065299 17247 sys_catalog.cc:263] Verifying existing consensus state
I20260925 19:08:28.066041 17247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c: Bootstrap starting.
I20260925 19:08:28.074805 17247 log.cc:824] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:28.077327 17247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:08:28.077670 17247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c: Bootstrap complete.
I20260925 19:08:28.080123 17247 raft_consensus.cc:348] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283208106c934ffd993371388e6acb4c" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41039 } }
I20260925 19:08:28.080427 17247 raft_consensus.cc:729] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283208106c934ffd993371388e6acb4c, State: Initialized, Role: FOLLOWER
I20260925 19:08:28.080631 17247 consensus_queue.cc:260] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283208106c934ffd993371388e6acb4c" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41039 } }
I20260925 19:08:28.080689 17247 raft_consensus.cc:388] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:08:28.080753 17247 raft_consensus.cc:482] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:08:28.080806 17247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [term 2 FOLLOWER]: Advancing to term 3
I20260925 19:08:28.081887 17247 raft_consensus.cc:504] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283208106c934ffd993371388e6acb4c" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41039 } }
I20260925 19:08:28.081988 17247 leader_election.cc:302] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 283208106c934ffd993371388e6acb4c; no voters:
I20260925 19:08:28.082239 17247 leader_election.cc:288] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 19:08:28.082269 17251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [term 3 FOLLOWER]: Leader election won for term 3
I20260925 19:08:28.082386 17251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [term 3 LEADER]: Becoming Leader. State: Replica: 283208106c934ffd993371388e6acb4c, State: Running, Role: LEADER
I20260925 19:08:28.082479 17251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283208106c934ffd993371388e6acb4c" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41039 } }
I20260925 19:08:28.082547 17247 sys_catalog.cc:564] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:08:28.082782 17252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "283208106c934ffd993371388e6acb4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283208106c934ffd993371388e6acb4c" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41039 } } }
I20260925 19:08:28.082885 17252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [sys.catalog]: This master's current role is: LEADER
I20260925 19:08:28.082783 17253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 283208106c934ffd993371388e6acb4c. Latest consensus state: current_term: 3 leader_uuid: "283208106c934ffd993371388e6acb4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283208106c934ffd993371388e6acb4c" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41039 } } }
I20260925 19:08:28.083218 17253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c [sys.catalog]: This master's current role is: LEADER
I20260925 19:08:28.083366 17255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:08:28.084203 17255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:08:28.084309 17255 catalog_manager.cc:1259] Loaded cluster ID: 3a354f1db28945af8e5b6ed8ba2b7cb0
I20260925 19:08:28.084357 17255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:08:28.085741 17255 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:08:28.085929 17255 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c: Loaded TSK: 0
I20260925 19:08:28.085981 17255 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 283208106c934ffd993371388e6acb4c: Loaded TSK: 1
F20260925 19:08:28.086220 17255 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5dac1ac9ad google::LogMessage::Fail() at ??:0
@ 0x7f5dac1af88a google::LogMessage::SendToLog() at ??:0
@ 0x7f5dac1ac4de google::LogMessage::Flush() at ??:0
@ 0x7f5dac1ada19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5dae919104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f5dae916d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f5db41061cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f5db410648e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f5db4106594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f5db40ccc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f5db40d6ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5dae07a0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5dae06f954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5daeed36db start_thread at ??:0
@ 0x7f5daacaa61f clone at ??:0
W20260925 19:08:28.168218 17249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:28.168443 17249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:28.168478 17249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:28.169782 17249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:28.169831 17249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:28.169862 17249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:08:28.171520 17249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:37989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:35643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=34401
--tserver_master_addrs=127.0.196.254:41039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:28.171800 17249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:28.172051 17249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:28.177469 17288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:28.177645 17249 server_base.cc:1034] running on GCE node
W20260925 19:08:28.177501 17291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:28.177476 17289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:28.177999 17249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:28.178309 17249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:28.179494 17249 hybrid_clock.cc:648] HybridClock initialized: now 1790363308179476 us; error 36 us; skew 500 ppm
Sep 25 19:08:28 dist-test-slave-rn98 krb5kdc[16107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:08:28.182353 17249 init.cc:377] Logged in from keytab as kudu/127.0.196.193@KRBTEST.COM (short username kudu)
I20260925 19:08:28.182909 17249 webserver.cc:460] Webserver started at http://127.0.196.193:34401/ using document root <none> and password file <none>
I20260925 19:08:28.183171 17249 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:28.183244 17249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:28.184744 17249 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:08:28.185523 17298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:28.185730 17249 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:08:28.185837 17249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "32b8f17051084c27a6ef472c2fbb312e"
format_stamp: "Formatted at 2026-09-25 19:07:46 on dist-test-slave-rn98"
server_key: "95bb34c2589216a0420e9cdce5fb14cd"
server_key_iv: "9e4d076f09bf6a566d12e28957e4dd2f"
server_key_version: "encryptionkey@0"
I20260925 19:08:28.186213 17249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:28.195791 17249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:28.198166 17249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:28.198344 17249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:28.198694 17249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:08:28.199167 17305 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 19:08:28.201398 17249 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 19:08:28.201542 17249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260925 19:08:28.201606 17249 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 19:08:28.202273 17249 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 19:08:28.202333 17249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 19:08:28.202414 17305 tablet_bootstrap.cc:492] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e: Bootstrap starting.
I20260925 19:08:28.210018 17249 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:35643
I20260925 19:08:28.210186 17412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:35643 every 8 connection(s)
I20260925 19:08:28.210501 17249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
I20260925 19:08:28.213814 17305 log.cc:824] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:28.216804 17305 tablet_bootstrap.cc:492] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:08:28.217149 17305 tablet_bootstrap.cc:492] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e: Bootstrap complete.
I20260925 19:08:28.218394 17305 ts_tablet_manager.cc:1397] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.005s
I20260925 19:08:28.219751 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 17249
I20260925 19:08:28.220052 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:43989
--local_ip_for_outbound_sockets=127.0.196.194
--tserver_master_addrs=127.0.196.254:41039
--webserver_port=37129
--webserver_interface=127.0.196.194
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:37989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:08:28.220062 17305 raft_consensus.cc:348] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:08:28.220412 17305 raft_consensus.cc:729] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32b8f17051084c27a6ef472c2fbb312e, State: Initialized, Role: FOLLOWER
I20260925 19:08:28.220597 17305 consensus_queue.cc:260] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:08:28.220872 17305 ts_tablet_manager.cc:1428] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e: Time spent starting tablet: real 0.002s user 0.006s sys 0.000s
W20260925 19:08:28.297374 17415 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:08:28.211299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.193:59519)
0925 19:08:28.211495 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:28.211542 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:28.212042 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:28.297145 (+ 85103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":118,"threads_started":1}
W20260925 19:08:28.297693 17413 heartbeater.cc:644] Failed to heartbeat to 127.0.196.254:41039 (0 consecutive failures): Network error: Failed to ping master at 127.0.196.254:41039: Client connection negotiation failed: client connection to 127.0.196.254:41039: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 19:08:28.298098 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:28.297968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.193:60445)
0925 19:08:28.298030 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:28.298077 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 19:08:28.298470 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:28.298370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.193:58883)
0925 19:08:28.298419 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:28.298443 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 19:08:28.298583 17413 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 19:08:28.307937 17418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:28.308175 17418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:28.308203 17418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:28.309478 17418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:28.309522 17418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:28.309549 17418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:08:28.311079 17418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:37989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:43989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=37129
--tserver_master_addrs=127.0.196.254:41039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:28.311286 17418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:28.311506 17418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:28.316933 17424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:28.316869 17423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:28.316869 17426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:28.317124 17418 server_base.cc:1034] running on GCE node
I20260925 19:08:28.317349 17418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:28.317659 17418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:28.318837 17418 hybrid_clock.cc:648] HybridClock initialized: now 1790363308318826 us; error 41 us; skew 500 ppm
Sep 25 19:08:28 dist-test-slave-rn98 krb5kdc[16107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:08:28.321929 17418 init.cc:377] Logged in from keytab as kudu/127.0.196.194@KRBTEST.COM (short username kudu)
I20260925 19:08:28.322458 17418 webserver.cc:460] Webserver started at http://127.0.196.194:37129/ using document root <none> and password file <none>
I20260925 19:08:28.322778 17418 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:28.322865 17418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:28.324654 17418 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:08:28.325714 17433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:28.325973 17418 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:08:28.326099 17418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "e8b6377bb8db4f038c6ae369d9906e80"
format_stamp: "Formatted at 2026-09-25 19:07:46 on dist-test-slave-rn98"
server_key: "5ac6036f557c209eee2b864d9e8c137e"
server_key_iv: "be22d3fe09633d6c66857d29c19f172f"
server_key_version: "encryptionkey@0"
I20260925 19:08:28.326534 17418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:28.347666 17418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:28.350162 17418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:28.350351 17418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:28.350736 17418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:08:28.351274 17440 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 19:08:28.352818 17418 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 19:08:28.352989 17418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 19:08:28.353040 17418 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 19:08:28.353704 17418 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 19:08:28.353773 17418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 19:08:28.353858 17440 tablet_bootstrap.cc:492] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80: Bootstrap starting.
I20260925 19:08:28.355796 17440 log.cc:824] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:28.359187 17440 tablet_bootstrap.cc:492] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:08:28.359524 17440 tablet_bootstrap.cc:492] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80: Bootstrap complete.
I20260925 19:08:28.359709 17440 ts_tablet_manager.cc:1397] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260925 19:08:28.360703 17418 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:43989
I20260925 19:08:28.360774 17548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:43989 every 8 connection(s)
I20260925 19:08:28.361086 17418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
I20260925 19:08:28.361416 17440 raft_consensus.cc:348] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:08:28.361757 17440 raft_consensus.cc:729] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8b6377bb8db4f038c6ae369d9906e80, State: Initialized, Role: FOLLOWER
I20260925 19:08:28.361914 17440 consensus_queue.cc:260] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:08:28.362217 17440 ts_tablet_manager.cc:1428] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20260925 19:08:28.362279 17551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:28.361871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.194:36909)
0925 19:08:28.362066 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:28.362158 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":112,"threads_started":1}
W20260925 19:08:28.362495 17549 heartbeater.cc:644] Failed to heartbeat to 127.0.196.254:41039 (0 consecutive failures): Network error: Failed to ping master at 127.0.196.254:41039: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
I20260925 19:08:28.362848 17551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:28.362745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.194:43403)
0925 19:08:28.362794 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:28.362818 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 19:08:28.363130 17551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:28.363046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.194:48525)
0925 19:08:28.363090 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:28.363111 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 19:08:28.363240 17549 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 19:08:28.366441 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 17418
I20260925 19:08:28.366755 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:43139
--local_ip_for_outbound_sockets=127.0.196.195
--tserver_master_addrs=127.0.196.254:41039
--webserver_port=40171
--webserver_interface=127.0.196.195
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:37989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:08:28.453059 17553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:28.453322 17553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:28.453349 17553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:28.454737 17553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:28.454792 17553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:28.454819 17553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:08:28.456432 17553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:37989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:43139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=40171
--tserver_master_addrs=127.0.196.254:41039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:28.456701 17553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:28.456972 17553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:28.462212 17558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:28.462188 17559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:28.462369 17553 server_base.cc:1034] running on GCE node
W20260925 19:08:28.462229 17561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:28.462985 17553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:28.463258 17553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:28.464419 17553 hybrid_clock.cc:648] HybridClock initialized: now 1790363308464431 us; error 55 us; skew 500 ppm
Sep 25 19:08:28 dist-test-slave-rn98 krb5kdc[16107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:08:28.467087 17553 init.cc:377] Logged in from keytab as kudu/127.0.196.195@KRBTEST.COM (short username kudu)
I20260925 19:08:28.467639 17553 webserver.cc:460] Webserver started at http://127.0.196.195:40171/ using document root <none> and password file <none>
I20260925 19:08:28.467890 17553 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:28.467962 17553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:28.469393 17553 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:08:28.470084 17568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:28.470273 17553 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:08:28.470381 17553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "b8ebe057555540bf8d65ba18ee36a54f"
format_stamp: "Formatted at 2026-09-25 19:07:46 on dist-test-slave-rn98"
server_key: "f5c943bf10f5c61d4cbc853dc01c4209"
server_key_iv: "bf0013e28242eb59c07c7651e107d25d"
server_key_version: "encryptionkey@0"
I20260925 19:08:28.470834 17553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:28.490943 17553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:28.493355 17553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:28.493568 17553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:28.493870 17553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:08:28.494231 17553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:08:28.494294 17553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:28.494343 17553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:08:28.494375 17553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:28.500958 17553 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:43139
I20260925 19:08:28.501051 17681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:43139 every 8 connection(s)
I20260925 19:08:28.501461 17553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
I20260925 19:08:28.502079 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 17553
I20260925 19:08:28.502825 17684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:28.502267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.195:41613)
0925 19:08:28.502476 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:28.502587 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":100,"threads_started":1}
W20260925 19:08:28.503239 17682 heartbeater.cc:644] Failed to heartbeat to 127.0.196.254:41039 (0 consecutive failures): Network error: Failed to ping master at 127.0.196.254:41039: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
I20260925 19:08:28.503541 17684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:28.503437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.195:43843)
0925 19:08:28.503489 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:28.503515 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 19:08:28.503811 17684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:28.503728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.195:52291)
0925 19:08:28.503771 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:28.503793 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 19:08:28.503949 17682 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 19:08:29.299322 17686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:29.299054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.193:48917)
0925 19:08:29.299194 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:29.299255 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260925 19:08:29.363917 17687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:29.363692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.194:48911)
0925 19:08:29.363830 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:29.363876 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":60,"threads_started":1}
I20260925 19:08:29.504530 17688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:29.504282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.195:57695)
0925 19:08:29.504435 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:29.504470 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":100,"threads_started":1}
I20260925 19:08:29.892560 17689 raft_consensus.cc:482] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:08:29.892690 17689 raft_consensus.cc:504] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:08:29.893051 17689 leader_election.cc:288] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e8b6377bb8db4f038c6ae369d9906e80 (127.0.196.194:43989), b8ebe057555540bf8d65ba18ee36a54f (127.0.196.195:43139)
Sep 25 19:08:29 dist-test-slave-rn98 krb5kdc[16107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for kudu/127.0.196.195@KRBTEST.COM
Sep 25 19:08:29 dist-test-slave-rn98 krb5kdc[16107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for kudu/127.0.196.194@KRBTEST.COM
I20260925 19:08:29.910069 17690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:29.893256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:43139 (local address 127.0.196.193:47891)
0925 19:08:29.893543 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:29.893553 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:29.893595 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:29.894367 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:29.894368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:29.894516 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:08:29.894580 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:29.894581 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:29.896357 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:29.896358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:29.897758 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:29.897762 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:29.898726 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:29.898727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:29.898891 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:29.903178 (+ 4287us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:08:29.903193 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:08:29.905299 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:08:29.908002 (+ 2703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:29.908012 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:29.908015 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:29.908349 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:29.908725 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:29.908726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:29.908726 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:29.908934 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:29.909435 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:08:29.909443 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:08:29.909592 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 19:08:29.909707 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 19:08:29.909956 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":91,"thread_start_us":89,"threads_started":1}
I20260925 19:08:29.910086 17692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:29.893339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:47497 (local address 127.0.196.194:43989)
0925 19:08:29.893503 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:29.893516 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:29.893546 (+ 30us) server_negotiation.cc:407] Connection header received
0925 19:08:29.893694 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:29.893696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:29.893753 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:29.893910 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:29.894737 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:29.896181 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:29.897878 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:29.898563 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:29.898731 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:29.905777 (+ 7046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:29.905782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:29.905782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:08:29.905794 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:29.908374 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:29.908838 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:29.908841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:29.908841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:29.908979 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:29.909179 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:29.909180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:29.909180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:29.909528 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:29.909640 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:29.909842 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 19:08:29.909973 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":59,"threads_started":1}
I20260925 19:08:29.910082 17693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:29.893381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:47891 (local address 127.0.196.195:43139)
0925 19:08:29.893546 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:29.893553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:29.893586 (+ 33us) server_negotiation.cc:407] Connection header received
0925 19:08:29.894197 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:29.894200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:29.894243 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:29.894354 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:29.894744 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:29.896182 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:29.897878 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:29.898568 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:29.898730 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:29.905513 (+ 6783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:29.905520 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:29.905521 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:08:29.905538 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:29.907878 (+ 2340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:29.908497 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:29.908502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:29.908502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:29.908626 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:29.909040 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:29.909041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:29.909042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:29.909359 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:29.909450 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:29.909787 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 19:08:29.909934 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":54,"threads_started":1}
I20260925 19:08:29.910210 17691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:29.893237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:43989 (local address 127.0.196.193:47497)
0925 19:08:29.893407 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:29.893421 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:29.893482 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:29.893951 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:29.893953 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:29.894258 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:08:29.894563 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:29.894574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:29.896322 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:29.896323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:29.897758 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:29.897762 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:29.898726 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:29.898727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:29.898891 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:29.903178 (+ 4287us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:08:29.903193 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:08:29.905652 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:08:29.908484 (+ 2832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:29.908485 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:29.908486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:29.908741 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:29.909054 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:29.909055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:29.909055 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:29.909113 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:29.909630 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:08:29.909631 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:08:29.909672 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 19:08:29.909768 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 19:08:29.910046 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":195,"thread_start_us":66,"threads_started":1}
I20260925 19:08:29.910989 17503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4082737c22b24a4aac98f891ebc5fb7c" candidate_uuid: "32b8f17051084c27a6ef472c2fbb312e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e8b6377bb8db4f038c6ae369d9906e80" is_pre_election: true
I20260925 19:08:29.911162 17503 raft_consensus.cc:2454] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32b8f17051084c27a6ef472c2fbb312e in term 1.
W20260925 19:08:29.911410 17302 leader_election.cc:341] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8ebe057555540bf8d65ba18ee36a54f (127.0.196.195:43139): Not found: Tablet not found: 4082737c22b24a4aac98f891ebc5fb7c
I20260925 19:08:29.911554 17300 leader_election.cc:302] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32b8f17051084c27a6ef472c2fbb312e, e8b6377bb8db4f038c6ae369d9906e80; no voters: b8ebe057555540bf8d65ba18ee36a54f
I20260925 19:08:29.911692 17689 raft_consensus.cc:2781] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 19:08:29.911756 17689 raft_consensus.cc:482] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:08:29.911787 17689 raft_consensus.cc:3037] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e [term 1 FOLLOWER]: Advancing to term 2
I20260925 19:08:29.912966 17689 raft_consensus.cc:504] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:08:29.913130 17689 leader_election.cc:288] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e [CANDIDATE]: Term 2 election: Requested vote from peers e8b6377bb8db4f038c6ae369d9906e80 (127.0.196.194:43989), b8ebe057555540bf8d65ba18ee36a54f (127.0.196.195:43139)
W20260925 19:08:29.913400 17302 leader_election.cc:341] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b8ebe057555540bf8d65ba18ee36a54f (127.0.196.195:43139): Not found: Tablet not found: 4082737c22b24a4aac98f891ebc5fb7c
I20260925 19:08:29.913335 17503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4082737c22b24a4aac98f891ebc5fb7c" candidate_uuid: "32b8f17051084c27a6ef472c2fbb312e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e8b6377bb8db4f038c6ae369d9906e80"
I20260925 19:08:29.913455 17503 raft_consensus.cc:3037] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80 [term 1 FOLLOWER]: Advancing to term 2
I20260925 19:08:29.914675 17503 raft_consensus.cc:2454] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32b8f17051084c27a6ef472c2fbb312e in term 2.
I20260925 19:08:29.914880 17300 leader_election.cc:302] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32b8f17051084c27a6ef472c2fbb312e, e8b6377bb8db4f038c6ae369d9906e80; no voters: b8ebe057555540bf8d65ba18ee36a54f
I20260925 19:08:29.914994 17689 raft_consensus.cc:2781] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e [term 2 FOLLOWER]: Leader election won for term 2
I20260925 19:08:29.915107 17689 raft_consensus.cc:686] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e [term 2 LEADER]: Becoming Leader. State: Replica: 32b8f17051084c27a6ef472c2fbb312e, State: Running, Role: LEADER
I20260925 19:08:29.915208 17689 consensus_queue.cc:237] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:08:29.915524 17690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:29.915354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.193:50747)
0925 19:08:29.915453 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:29.915485 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260925 19:08:30.364779 17696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:30.364486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.194:50923)
0925 19:08:30.364683 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:30.364731 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":87,"threads_started":1}
I20260925 19:08:30.505260 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:30.505034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.195:42515)
0925 19:08:30.505158 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:30.505201 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260925 19:08:30.523128 17503 raft_consensus.cc:1261] T 4082737c22b24a4aac98f891ebc5fb7c P e8b6377bb8db4f038c6ae369d9906e80 [term 2 FOLLOWER]: Refusing update from remote peer 32b8f17051084c27a6ef472c2fbb312e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260925 19:08:30.523633 17698 consensus_queue.cc:1035] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 19:08:30.524758 17696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:30.524523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.194:43667)
0925 19:08:30.524597 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:30.524687 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 19:08:30.525694 17696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:30.525598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.194:55179)
0925 19:08:30.525637 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:30.525663 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 19:08:30.525681 17302 consensus_peers.cc:487] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e -> Peer b8ebe057555540bf8d65ba18ee36a54f (127.0.196.195:43139): Couldn't send request to peer b8ebe057555540bf8d65ba18ee36a54f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4082737c22b24a4aac98f891ebc5fb7c. This is attempt 1: this message will repeat every 5th retry.
I20260925 19:08:30.525959 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:30.525699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.193:46469)
0925 19:08:30.525888 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:30.525932 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":103,"threads_started":1}
I20260925 19:08:30.969560 17706 ts_tablet_manager.cc:927] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f: Initiating tablet copy from peer 32b8f17051084c27a6ef472c2fbb312e (127.0.196.193:35643)
I20260925 19:08:30.969823 17706 tablet_copy_client.cc:316] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.196.193:35643
Sep 25 19:08:30 dist-test-slave-rn98 krb5kdc[16107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for kudu/127.0.196.193@KRBTEST.COM
I20260925 19:08:30.978377 17708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:30.970250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:39283 (local address 127.0.196.193:35643)
0925 19:08:30.970444 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:30.970449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:30.970460 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:08:30.970485 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:30.970488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:30.970500 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:30.970564 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:30.971107 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:30.971816 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:30.972669 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:30.973003 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:30.973087 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:30.975499 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:30.975511 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:30.975512 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:08:30.975523 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:30.977016 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:30.977450 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:30.977458 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:30.977458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:30.977513 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:30.977729 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:30.977730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:30.977730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:30.977887 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:30.977979 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:30.978172 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 19:08:30.978261 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":113,"threads_started":1}
I20260925 19:08:30.978395 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:30.970116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:35643 (local address 127.0.196.195:39283)
0925 19:08:30.970234 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:30.970259 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:30.970355 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:30.970593 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:30.970597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:30.970961 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:08:30.971025 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:30.971027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:30.971953 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:30.971955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:30.972573 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:30.972575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:30.973087 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:30.973088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:30.973177 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:30.973796 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:08:30.973801 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:08:30.975393 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:08:30.977114 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:30.977122 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:30.977124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:30.977338 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:30.977593 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:30.977594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:30.977594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:30.977658 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:30.977970 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:08:30.977977 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:08:30.978035 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 19:08:30.978137 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 19:08:30.978261 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 19:08:30.978964 17387 tablet_copy_service.cc:140] P 32b8f17051084c27a6ef472c2fbb312e: Received BeginTabletCopySession request for tablet 4082737c22b24a4aac98f891ebc5fb7c from peer b8ebe057555540bf8d65ba18ee36a54f ({username='kudu', principal='kudu/127.0.196.195@KRBTEST.COM'} at 127.0.196.195:39283)
I20260925 19:08:30.979029 17387 tablet_copy_service.cc:161] P 32b8f17051084c27a6ef472c2fbb312e: Beginning new tablet copy session on tablet 4082737c22b24a4aac98f891ebc5fb7c from peer b8ebe057555540bf8d65ba18ee36a54f at {username='kudu', principal='kudu/127.0.196.195@KRBTEST.COM'} at 127.0.196.195:39283: session id = b8ebe057555540bf8d65ba18ee36a54f-4082737c22b24a4aac98f891ebc5fb7c
I20260925 19:08:30.979517 17387 tablet_copy_source_session.cc:205] T 4082737c22b24a4aac98f891ebc5fb7c P 32b8f17051084c27a6ef472c2fbb312e: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 19:08:30.980499 17706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4082737c22b24a4aac98f891ebc5fb7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:30.983738 17706 tablet_copy_client.cc:791] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f: tablet copy: Starting download of 0 data blocks...
I20260925 19:08:30.983918 17706 tablet_copy_client.cc:655] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f: tablet copy: Starting download of 1 WAL segments...
I20260925 19:08:30.984966 17706 tablet_copy_client.cc:531] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 19:08:30.985726 17706 tablet_bootstrap.cc:492] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f: Bootstrap starting.
I20260925 19:08:30.987265 17706 log.cc:824] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:30.990392 17706 tablet_bootstrap.cc:492] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:08:30.990609 17706 tablet_bootstrap.cc:492] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f: Bootstrap complete.
I20260925 19:08:30.990809 17706 ts_tablet_manager.cc:1397] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f: Time spent bootstrapping tablet: real 0.005s user 0.006s sys 0.000s
I20260925 19:08:30.991853 17706 raft_consensus.cc:348] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:08:30.992018 17706 raft_consensus.cc:729] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ebe057555540bf8d65ba18ee36a54f, State: Initialized, Role: FOLLOWER
I20260925 19:08:30.992161 17706 consensus_queue.cc:260] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b6377bb8db4f038c6ae369d9906e80" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 43989 } } peers { permanent_uuid: "b8ebe057555540bf8d65ba18ee36a54f" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 43139 } } peers { permanent_uuid: "32b8f17051084c27a6ef472c2fbb312e" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 35643 } }
I20260925 19:08:30.992436 17706 ts_tablet_manager.cc:1428] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:08:30.992775 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:30.992666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.195:37867)
0925 19:08:30.992705 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:30.992735 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260925 19:08:30.992791 17387 tablet_copy_service.cc:342] P 32b8f17051084c27a6ef472c2fbb312e: Request end of tablet copy session b8ebe057555540bf8d65ba18ee36a54f-4082737c22b24a4aac98f891ebc5fb7c received from {username='kudu', principal='kudu/127.0.196.195@KRBTEST.COM'} at 127.0.196.195:39283
I20260925 19:08:30.992873 17387 tablet_copy_service.cc:433] P 32b8f17051084c27a6ef472c2fbb312e: ending tablet copy session b8ebe057555540bf8d65ba18ee36a54f-4082737c22b24a4aac98f891ebc5fb7c on tablet 4082737c22b24a4aac98f891ebc5fb7c with peer b8ebe057555540bf8d65ba18ee36a54f
I20260925 19:08:30.993117 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:30.993029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.195:52951)
0925 19:08:30.993067 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:30.993092 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 19:08:31.426771 17636 raft_consensus.cc:1203] T 4082737c22b24a4aac98f891ebc5fb7c P b8ebe057555540bf8d65ba18ee36a54f [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260925 19:08:31.427217 17713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:31.426713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.193:36585)
0925 19:08:31.427101 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:31.427166 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":37,"threads_started":1}
I20260925 19:08:31.427248 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:31.427132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.195:33455)
0925 19:08:31.427181 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:31.427206 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 19:08:31.526300 17714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:31.526077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.194:42993)
0925 19:08:31.526206 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:31.526252 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260925 19:08:32.428138 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:32.427868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.195:48733)
0925 19:08:32.428022 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:32.428079 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
I20260925 19:08:32.428136 17715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:32.427868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.193:58185)
0925 19:08:32.428048 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:32.428090 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":82,"threads_started":1}
I20260925 19:08:32.527153 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:32.526871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.194:43397)
0925 19:08:32.527065 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:32.527106 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":79,"threads_started":1}
I20260925 19:08:33.429066 17719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:33.428851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.195:33283)
0925 19:08:33.428980 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:33.429018 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260925 19:08:33.429112 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:33.428845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.193:54269)
0925 19:08:33.428972 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:33.429058 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260925 19:08:33.429489 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:33.429383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.193:38303)
0925 19:08:33.429434 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:33.429458 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 19:08:33.528024 17720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:33.527761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.194:41373)
0925 19:08:33.527928 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:33.527977 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":70,"threads_started":1}
I20260925 19:08:34.429908 17721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:34.429599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.195:51379)
0925 19:08:34.429805 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:34.429859 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":123,"threads_started":1}
I20260925 19:08:34.430167 17722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:34.429940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.193:45467)
0925 19:08:34.430093 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:34.430133 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":76,"threads_started":1}
I20260925 19:08:34.529043 17723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:34.528730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.194:34717)
0925 19:08:34.528945 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:34.528989 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":83,"threads_started":1}
I20260925 19:08:35.430696 17724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:35.430403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.195:44809)
0925 19:08:35.430552 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:35.430589 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260925 19:08:35.430948 17725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:35.430682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.193:35799)
0925 19:08:35.430810 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:35.430881 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260925 19:08:35.529927 17726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:35.529685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.194:44667)
0925 19:08:35.529833 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:35.529880 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":64,"threads_started":1}
I20260925 19:08:36.431547 17727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:36.431294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.195:59317)
0925 19:08:36.431437 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:36.431476 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260925 19:08:36.431680 17728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:36.431420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.193:33469)
0925 19:08:36.431578 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:36.431614 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":62,"threads_started":1}
I20260925 19:08:36.530730 17729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:36.530431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.194:48229)
0925 19:08:36.530580 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:36.530681 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":67,"threads_started":1}
I20260925 19:08:37.432377 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:37.432147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.195:52255)
0925 19:08:37.432285 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:37.432329 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260925 19:08:37.432377 17731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:37.432146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.193:36159)
0925 19:08:37.432289 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:37.432329 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260925 19:08:37.531499 17732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:37.531265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.194:36825)
0925 19:08:37.531413 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:37.531449 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":65,"threads_started":1}
I20260925 19:08:38.433271 17733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:38.433025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.195:55653)
0925 19:08:38.433175 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:38.433221 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260925 19:08:38.433287 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:38.433025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.193:47409)
0925 19:08:38.433212 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:38.433246 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":111,"threads_started":1}
I20260925 19:08:38.532357 17735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:38.532107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.194:40767)
0925 19:08:38.532256 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:38.532294 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260925 19:08:39.434077 17736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:39.433836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.193:54545)
0925 19:08:39.433976 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:39.434023 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260925 19:08:39.434084 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:39.433836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.195:47635)
0925 19:08:39.434010 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:39.434042 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":80,"threads_started":1}
I20260925 19:08:39.533102 17738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:39.532849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.194:47911)
0925 19:08:39.532993 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:39.533033 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260925 19:08:40.434777 17739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:40.434514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.193:56667)
0925 19:08:40.434685 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:40.434728 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":68,"threads_started":1}
I20260925 19:08:40.434811 17740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:40.434571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.195:47169)
0925 19:08:40.434730 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:40.434768 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":94,"threads_started":1}
I20260925 19:08:40.533986 17741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:40.533711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.194:34971)
0925 19:08:40.533875 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:40.533924 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":71,"threads_started":1}
I20260925 19:08:41.435527 17742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:41.435253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.193:54649)
0925 19:08:41.435419 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:41.435470 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":85,"threads_started":1}
I20260925 19:08:41.435572 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:41.435312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.195:43805)
0925 19:08:41.435488 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:41.435528 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":88,"threads_started":1}
I20260925 19:08:41.534835 17744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:41.534558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.194:50611)
0925 19:08:41.534735 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:41.534782 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":111,"threads_started":1}
I20260925 19:08:42.436316 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:42.436064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.193:53713)
0925 19:08:42.436231 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:42.436269 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":107,"threads_started":1}
I20260925 19:08:42.436358 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:42.436138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.195:52569)
0925 19:08:42.436283 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:42.436316 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
I20260925 19:08:42.535621 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:42.535376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.194:42907)
0925 19:08:42.535524 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:42.535572 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":67,"threads_started":1}
I20260925 19:08:43.437126 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:43.436887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.193:45059)
0925 19:08:43.437038 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:43.437081 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":54,"threads_started":1}
I20260925 19:08:43.437126 17749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:43.436894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41039 (local address 127.0.196.195:46639)
0925 19:08:43.437042 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:43.437081 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.254:41039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
I20260925 19:08:43.511518 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 17249
I20260925 19:08:43.516749 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 17418
I20260925 19:08:43.521559 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 17553
2026-09-25T19:08:43Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58538 ms)
[----------] 30 tests from SecurityITest (236443 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:08:43 dist-test-slave-rn98 krb5kdc[17752](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:08:43 dist-test-slave-rn98 krb5kdc[17752](info): set up 2 sockets
Sep 25 19:08:43 dist-test-slave-rn98 krb5kdc[17752](info): commencing operation
krb5kdc: starting...
W20260925 19:08:44.574879 787 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:08:44 dist-test-slave-rn98 krb5kdc[17752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:08:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:08:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:08:44.701304 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:35921
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:38053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:35921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:08:44.784659 17765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:44.784915 17765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:44.784940 17765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:44.786244 17765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:08:44.786299 17765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:44.786314 17765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:44.786327 17765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:08:44.786340 17765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:08:44.788007 17765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:38053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:35921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:35921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:44.788298 17765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:44.788551 17765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:44.794291 17770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:44.794265 17773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:44.794340 17765 server_base.cc:1034] running on GCE node
W20260925 19:08:44.794265 17771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:44.794831 17765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:44.795161 17765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:44.796335 17765 hybrid_clock.cc:648] HybridClock initialized: now 1790363324796320 us; error 39 us; skew 500 ppm
Sep 25 19:08:44 dist-test-slave-rn98 krb5kdc[17752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:08:44.799182 17765 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:08:44.799624 17765 webserver.cc:460] Webserver started at http://127.0.196.254:46323/ using document root <none> and password file <none>
I20260925 19:08:44.799923 17765 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:44.799973 17765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:44.800132 17765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:44.801265 17765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "a4677ec3b29e44c2862628476f21165a"
format_stamp: "Formatted at 2026-09-25 19:08:44 on dist-test-slave-rn98"
server_key: "6b961e5c07b8ab051fb563bb2fe9cb46"
server_key_iv: "728dfab08992c2e72f6b3106097e5fb2"
server_key_version: "encryptionkey@0"
I20260925 19:08:44.801662 17765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "a4677ec3b29e44c2862628476f21165a"
format_stamp: "Formatted at 2026-09-25 19:08:44 on dist-test-slave-rn98"
server_key: "6b961e5c07b8ab051fb563bb2fe9cb46"
server_key_iv: "728dfab08992c2e72f6b3106097e5fb2"
server_key_version: "encryptionkey@0"
I20260925 19:08:44.803200 17765 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:08:44.804054 17780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:44.804239 17765 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:08:44.804327 17765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "a4677ec3b29e44c2862628476f21165a"
format_stamp: "Formatted at 2026-09-25 19:08:44 on dist-test-slave-rn98"
server_key: "6b961e5c07b8ab051fb563bb2fe9cb46"
server_key_iv: "728dfab08992c2e72f6b3106097e5fb2"
server_key_version: "encryptionkey@0"
I20260925 19:08:44.804389 17765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:44.833051 17765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:44.835562 17765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:44.835695 17765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:44.839737 17765 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:35921
I20260925 19:08:44.839778 17832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:35921 every 8 connection(s)
I20260925 19:08:44.840183 17765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:08:44.841241 17833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:44.844116 17833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4677ec3b29e44c2862628476f21165a: Bootstrap starting.
I20260925 19:08:44.844743 17833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4677ec3b29e44c2862628476f21165a: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:44.845006 17833 log.cc:824] T 00000000000000000000000000000000 P a4677ec3b29e44c2862628476f21165a: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:44.845784 17833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4677ec3b29e44c2862628476f21165a: No bootstrap required, opened a new log
I20260925 19:08:44.847078 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 17765
I20260925 19:08:44.847169 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:08:44.847388 787 external_mini_cluster.cc:1428] Setting key 41bc34762d92812f359f499105c3e16c
I20260925 19:08:44.847858 17833 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4677ec3b29e44c2862628476f21165a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4677ec3b29e44c2862628476f21165a" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 35921 } }
I20260925 19:08:44.848091 17833 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4677ec3b29e44c2862628476f21165a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:44.848138 17833 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4677ec3b29e44c2862628476f21165a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4677ec3b29e44c2862628476f21165a, State: Initialized, Role: FOLLOWER
I20260925 19:08:44.848297 17833 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4677ec3b29e44c2862628476f21165a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4677ec3b29e44c2862628476f21165a" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 35921 } }
I20260925 19:08:44.848405 17833 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4677ec3b29e44c2862628476f21165a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:08:44.848448 17833 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4677ec3b29e44c2862628476f21165a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:08:44.848502 17833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4677ec3b29e44c2862628476f21165a [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:44.849713 17833 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4677ec3b29e44c2862628476f21165a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4677ec3b29e44c2862628476f21165a" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 35921 } }
I20260925 19:08:44.849841 17833 leader_election.cc:302] T 00000000000000000000000000000000 P a4677ec3b29e44c2862628476f21165a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4677ec3b29e44c2862628476f21165a; no voters:
I20260925 19:08:44.850036 17833 leader_election.cc:288] T 00000000000000000000000000000000 P a4677ec3b29e44c2862628476f21165a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:08:44.850059 17838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4677ec3b29e44c2862628476f21165a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:08:44.850171 17838 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4677ec3b29e44c2862628476f21165a [term 1 LEADER]: Becoming Leader. State: Replica: a4677ec3b29e44c2862628476f21165a, State: Running, Role: LEADER
I20260925 19:08:44.850274 17838 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4677ec3b29e44c2862628476f21165a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4677ec3b29e44c2862628476f21165a" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 35921 } }
I20260925 19:08:44.850320 17833 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4677ec3b29e44c2862628476f21165a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:08:44.851225 17839 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4677ec3b29e44c2862628476f21165a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4677ec3b29e44c2862628476f21165a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4677ec3b29e44c2862628476f21165a" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 35921 } } }
I20260925 19:08:44.851250 17840 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4677ec3b29e44c2862628476f21165a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4677ec3b29e44c2862628476f21165a. Latest consensus state: current_term: 1 leader_uuid: "a4677ec3b29e44c2862628476f21165a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4677ec3b29e44c2862628476f21165a" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 35921 } } }
I20260925 19:08:44.851352 17840 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4677ec3b29e44c2862628476f21165a [sys.catalog]: This master's current role is: LEADER
I20260925 19:08:44.851310 17839 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4677ec3b29e44c2862628476f21165a [sys.catalog]: This master's current role is: LEADER
I20260925 19:08:44.851665 17849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:08:44.852520 17849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:08:44.854146 17849 catalog_manager.cc:1347] Generated new cluster ID: 15f72ea9b25c4d85b5f7a9dc7a2a569e
I20260925 19:08:44.854199 17849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:08:44 dist-test-slave-rn98 krb5kdc[17752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:08:44.860263 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:44.847966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45942 (local address 127.0.196.254:35921)
0925 19:08:44.848220 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:44.848229 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:44.848254 (+ 25us) server_negotiation.cc:407] Connection header received
0925 19:08:44.848392 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:44.848398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:44.848475 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:44.848622 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:44.849437 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:44.850644 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:44.851778 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:44.852423 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:44.852550 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:44.855507 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:44.855512 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:44.855512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:08:44.855526 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:44.857706 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:44.858162 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:44.858166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:44.858167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:44.858314 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:44.858530 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:44.858530 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:44.858531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:44.858917 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:44.859071 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:44.859624 (+ 553us) server_negotiation.cc:299] Negotiation successful
0925 19:08:44.859953 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":108,"threads_started":1}
I20260925 19:08:44.867817 17849 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:08:44.868587 17849 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:08:44.873929 17849 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4677ec3b29e44c2862628476f21165a: Generated new TSK 0
I20260925 19:08:44.874174 17849 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
I20260925 19:08:44.902034 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:0
--local_ip_for_outbound_sockets=127.0.196.193
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:35921
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:38053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:08:44.988452 17859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:44.988713 17859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:44.988754 17859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:44.990769 17859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:44.990900 17859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:44.990978 17859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:08:44.992707 17859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:38053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:35921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:44.992993 17859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:44.993250 17859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:44.998732 17865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:44.998718 17864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:44.998735 17867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:44.999056 17859 server_base.cc:1034] running on GCE node
I20260925 19:08:44.999303 17859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:44.999574 17859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:45.000754 17859 hybrid_clock.cc:648] HybridClock initialized: now 1790363325000740 us; error 36 us; skew 500 ppm
Sep 25 19:08:45 dist-test-slave-rn98 krb5kdc[17752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:08:45.003568 17859 init.cc:377] Logged in from keytab as kudu/127.0.196.193@KRBTEST.COM (short username kudu)
I20260925 19:08:45.004046 17859 webserver.cc:460] Webserver started at http://127.0.196.193:33079/ using document root <none> and password file <none>
I20260925 19:08:45.004280 17859 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:45.004347 17859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:45.004495 17859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:45.005576 17859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-0/data/instance:
uuid: "ac4ee18cd64440b39bec2381b4e565b8"
format_stamp: "Formatted at 2026-09-25 19:08:45 on dist-test-slave-rn98"
server_key: "aa48443517d27a0117a02fac906e913a"
server_key_iv: "8b9b8eb3c1ac20216e5d0488ce26d00f"
server_key_version: "encryptionkey@0"
I20260925 19:08:45.005988 17859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-0/wal/instance:
uuid: "ac4ee18cd64440b39bec2381b4e565b8"
format_stamp: "Formatted at 2026-09-25 19:08:45 on dist-test-slave-rn98"
server_key: "aa48443517d27a0117a02fac906e913a"
server_key_iv: "8b9b8eb3c1ac20216e5d0488ce26d00f"
server_key_version: "encryptionkey@0"
I20260925 19:08:45.007519 17859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:08:45.008309 17874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:45.008486 17859 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:08:45.008585 17859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "ac4ee18cd64440b39bec2381b4e565b8"
format_stamp: "Formatted at 2026-09-25 19:08:45 on dist-test-slave-rn98"
server_key: "aa48443517d27a0117a02fac906e913a"
server_key_iv: "8b9b8eb3c1ac20216e5d0488ce26d00f"
server_key_version: "encryptionkey@0"
I20260925 19:08:45.008665 17859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:45.020485 17859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:45.023069 17859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:45.023257 17859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:45.023532 17859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:08:45.023886 17859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:08:45.023942 17859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:45.023981 17859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:08:45.024026 17859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:45.030963 17859 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:45393
I20260925 19:08:45.031018 17987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:45393 every 8 connection(s)
I20260925 19:08:45.031387 17859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
I20260925 19:08:45.037868 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 17859
I20260925 19:08:45.038004 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-0/wal/instance
Sep 25 19:08:45 dist-test-slave-rn98 krb5kdc[17752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:08:45.038241 787 external_mini_cluster.cc:1428] Setting key 80626e1f3df8502b3d8a0586ba44bb10
I20260925 19:08:45.041890 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:45.032205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:52219 (local address 127.0.196.254:35921)
0925 19:08:45.032288 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:45.032290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:45.032554 (+ 264us) server_negotiation.cc:407] Connection header received
0925 19:08:45.033513 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:45.033514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:45.033529 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:45.033594 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:45.034318 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.034905 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.035932 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.036166 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.036237 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:45.038667 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:45.038668 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:45.038669 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:08:45.038685 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:45.040365 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:45.040811 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:45.040812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:45.040813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:45.040873 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:45.041136 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:45.041137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:45.041137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:45.041260 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:45.041335 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:45.041620 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 19:08:45.041724 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 19:08:45.042023 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:45.032165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:35921 (local address 127.0.196.193:52219)
0925 19:08:45.032390 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:45.032433 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:45.033403 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:45.033709 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:45.033713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:45.034005 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:08:45.034210 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:45.034214 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.034985 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.034985 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:45.035821 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:45.035823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.036252 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.036253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:45.036362 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:45.036955 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:08:45.036965 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:08:45.038534 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:08:45.040454 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:45.040462 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:45.040465 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:45.040716 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:45.040947 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:45.040948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:45.040948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:45.041070 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:45.041327 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:08:45.041335 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:08:45.041486 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 19:08:45.041597 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 19:08:45.041776 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":103,"threads_started":1}
I20260925 19:08:45.042546 17988 heartbeater.cc:347] Connected to a master server at 127.0.196.254:35921
I20260925 19:08:45.042685 17988 heartbeater.cc:464] Registering TS with master...
I20260925 19:08:45.042886 17988 heartbeater.cc:511] Master 127.0.196.254:35921 requested a full tablet report, sending...
I20260925 19:08:45.043431 17797 ts_manager.cc:194] Registered new tserver with Master: ac4ee18cd64440b39bec2381b4e565b8 (127.0.196.193:45393)
I20260925 19:08:45.044055 17797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.193@KRBTEST.COM'} at 127.0.196.193:52219
WARNING: no policy specified for kudu/127.0.196.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
I20260925 19:08:45.066836 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:0
--local_ip_for_outbound_sockets=127.0.196.194
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:35921
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:38053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:08:45.149672 17993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:45.149961 17993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:45.150007 17993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:45.151512 17993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:45.151580 17993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:45.151630 17993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:08:45.153255 17993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:38053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:35921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:45.153555 17993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:45.153815 17993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:45.159129 17998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:45.159197 18001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:45.159132 17999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:45.159410 17993 server_base.cc:1034] running on GCE node
I20260925 19:08:45.159641 17993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:45.159879 17993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:45.161032 17993 hybrid_clock.cc:648] HybridClock initialized: now 1790363325161017 us; error 28 us; skew 500 ppm
Sep 25 19:08:45 dist-test-slave-rn98 krb5kdc[17752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:08:45.163733 17993 init.cc:377] Logged in from keytab as kudu/127.0.196.194@KRBTEST.COM (short username kudu)
I20260925 19:08:45.164258 17993 webserver.cc:460] Webserver started at http://127.0.196.194:33553/ using document root <none> and password file <none>
I20260925 19:08:45.164508 17993 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:45.164574 17993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:45.164717 17993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:45.165808 17993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-1/data/instance:
uuid: "a5fb57d4c98b4126a9b517a5da9c1b7a"
format_stamp: "Formatted at 2026-09-25 19:08:45 on dist-test-slave-rn98"
server_key: "37e1414a17f6a0213e1753bda0fec5bc"
server_key_iv: "58203a6aea26881c1e6461cd11c21dcf"
server_key_version: "encryptionkey@0"
I20260925 19:08:45.166191 17993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-1/wal/instance:
uuid: "a5fb57d4c98b4126a9b517a5da9c1b7a"
format_stamp: "Formatted at 2026-09-25 19:08:45 on dist-test-slave-rn98"
server_key: "37e1414a17f6a0213e1753bda0fec5bc"
server_key_iv: "58203a6aea26881c1e6461cd11c21dcf"
server_key_version: "encryptionkey@0"
I20260925 19:08:45.167639 17993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:08:45.168462 18008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:45.168645 17993 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:08:45.168749 17993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "a5fb57d4c98b4126a9b517a5da9c1b7a"
format_stamp: "Formatted at 2026-09-25 19:08:45 on dist-test-slave-rn98"
server_key: "37e1414a17f6a0213e1753bda0fec5bc"
server_key_iv: "58203a6aea26881c1e6461cd11c21dcf"
server_key_version: "encryptionkey@0"
I20260925 19:08:45.168828 17993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:45.181567 17993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:45.184069 17993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:45.184248 17993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:45.184511 17993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:08:45.184836 17993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:08:45.184886 17993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:45.184921 17993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:08:45.184969 17993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:45.191217 17993 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:45379
I20260925 19:08:45.191263 18121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:45379 every 8 connection(s)
I20260925 19:08:45.191632 17993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
I20260925 19:08:45.192118 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 17993
I20260925 19:08:45.192224 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-1/wal/instance
I20260925 19:08:45.192431 787 external_mini_cluster.cc:1428] Setting key 1dcb6b603ddc8a0b143d79978ad4ef96
Sep 25 19:08:45 dist-test-slave-rn98 krb5kdc[17752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.196.195@KRBTEST.COM; defaulting to no policy
I20260925 19:08:45.202553 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:45.192430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:60821 (local address 127.0.196.254:35921)
0925 19:08:45.192490 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:45.192492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:45.192801 (+ 309us) server_negotiation.cc:407] Connection header received
0925 19:08:45.193842 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:45.193843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:45.193856 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:45.193911 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:45.194602 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.195259 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.196312 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.196543 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.196633 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:45.199126 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:45.199128 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:45.199128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:08:45.199137 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:45.200887 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:45.201362 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:45.201363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:45.201364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:45.201425 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:45.201733 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:45.201734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:45.201734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:45.201901 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:45.201981 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:45.202322 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 19:08:45.202453 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:08:45.202731 18124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:45.192444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:35921 (local address 127.0.196.194:60821)
0925 19:08:45.192651 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:45.192699 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:45.193721 (+ 1022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:45.194013 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:45.194018 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:45.194296 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:08:45.194506 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:45.194510 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.195368 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.195370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:45.196220 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:45.196222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.196641 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.196642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:45.196774 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:45.197373 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:08:45.197381 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:08:45.199021 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:08:45.200974 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:45.200982 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:45.200985 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:45.201223 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:45.201527 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:45.201528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:45.201528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:45.201660 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:45.201971 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:08:45.201979 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:08:45.202145 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 19:08:45.202254 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 19:08:45.202457 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":83,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.195@KRBTEST.COM" created.
I20260925 19:08:45.203269 18122 heartbeater.cc:347] Connected to a master server at 127.0.196.254:35921
I20260925 19:08:45.203395 18122 heartbeater.cc:464] Registering TS with master...
I20260925 19:08:45.203642 18122 heartbeater.cc:511] Master 127.0.196.254:35921 requested a full tablet report, sending...
I20260925 19:08:45.204069 17797 ts_manager.cc:194] Registered new tserver with Master: a5fb57d4c98b4126a9b517a5da9c1b7a (127.0.196.194:45379)
I20260925 19:08:45.204512 17797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.194@KRBTEST.COM'} at 127.0.196.194:60821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
I20260925 19:08:45.219529 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:0
--local_ip_for_outbound_sockets=127.0.196.195
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:35921
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:38053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:08:45.306893 18127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:45.307138 18127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:45.307165 18127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:45.308404 18127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:45.308450 18127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:45.308478 18127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:08:45.309968 18127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:38053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:35921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:45.310163 18127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:45.310351 18127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:45.315551 18132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:45.315560 18135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:45.315593 18133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:45.315852 18127 server_base.cc:1034] running on GCE node
I20260925 19:08:45.316071 18127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:45.316339 18127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:45.317544 18127 hybrid_clock.cc:648] HybridClock initialized: now 1790363325317531 us; error 30 us; skew 500 ppm
Sep 25 19:08:45 dist-test-slave-rn98 krb5kdc[17752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:08:45.320238 18127 init.cc:377] Logged in from keytab as kudu/127.0.196.195@KRBTEST.COM (short username kudu)
I20260925 19:08:45.320760 18127 webserver.cc:460] Webserver started at http://127.0.196.195:37611/ using document root <none> and password file <none>
I20260925 19:08:45.321017 18127 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:45.321084 18127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:45.321237 18127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:45.322321 18127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-2/data/instance:
uuid: "898332ff3e624b0c8e8482382ee89857"
format_stamp: "Formatted at 2026-09-25 19:08:45 on dist-test-slave-rn98"
server_key: "fbbf3f5f18397d9f4065a81f40e4ee8f"
server_key_iv: "1c25f27beca74bec4cb0e822887997a7"
server_key_version: "encryptionkey@0"
I20260925 19:08:45.322767 18127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-2/wal/instance:
uuid: "898332ff3e624b0c8e8482382ee89857"
format_stamp: "Formatted at 2026-09-25 19:08:45 on dist-test-slave-rn98"
server_key: "fbbf3f5f18397d9f4065a81f40e4ee8f"
server_key_iv: "1c25f27beca74bec4cb0e822887997a7"
server_key_version: "encryptionkey@0"
I20260925 19:08:45.324220 18127 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:08:45.325230 18142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:45.325446 18127 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:08:45.325541 18127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "898332ff3e624b0c8e8482382ee89857"
format_stamp: "Formatted at 2026-09-25 19:08:45 on dist-test-slave-rn98"
server_key: "fbbf3f5f18397d9f4065a81f40e4ee8f"
server_key_iv: "1c25f27beca74bec4cb0e822887997a7"
server_key_version: "encryptionkey@0"
I20260925 19:08:45.325630 18127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:45.352372 18127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:45.354817 18127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:45.354988 18127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:45.355253 18127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:08:45.355603 18127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:08:45.355657 18127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:45.355695 18127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:08:45.355720 18127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:45.361894 18127 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:35303
I20260925 19:08:45.361939 18255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:35303 every 8 connection(s)
I20260925 19:08:45.362272 18127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
I20260925 19:08:45.365404 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 18127
I20260925 19:08:45.365512 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-2/wal/instance
I20260925 19:08:45.365690 787 external_mini_cluster.cc:1428] Setting key d1951575321357b56a4f82356acec4a5
Sep 25 19:08:45 dist-test-slave-rn98 krb5kdc[17752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:08:45.372512 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:45.363008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:40225 (local address 127.0.196.254:35921)
0925 19:08:45.363096 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:45.363100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:45.363296 (+ 196us) server_negotiation.cc:407] Connection header received
0925 19:08:45.364237 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:45.364239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:45.364249 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:45.364307 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:45.364939 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.365425 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.366910 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.367126 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.367223 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:45.369506 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:45.369508 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:45.369509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:08:45.369521 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:45.371002 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:45.371443 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:45.371444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:45.371445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:45.371505 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:45.371782 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:45.371784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:45.371784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:45.371973 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:45.372041 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:45.372317 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 19:08:45.372403 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 19:08:45.372697 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:45.362985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:35921 (local address 127.0.196.195:40225)
0925 19:08:45.363166 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:45.363205 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:45.364135 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:45.364370 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:45.364374 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:45.364659 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:08:45.364863 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:45.364868 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.365540 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.365541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:45.366825 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:45.366828 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.367231 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.367232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:45.367389 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:45.367967 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:08:45.367976 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:08:45.369393 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:08:45.371106 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:45.371114 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:45.371117 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:45.371357 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:45.371571 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:45.371572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:45.371572 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:45.371696 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:45.372039 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:08:45.372044 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:08:45.372169 (+ 125us) client_negotiation.cc:769] Sending connection context
0925 19:08:45.372279 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 19:08:45.372451 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":105,"threads_started":1}
I20260925 19:08:45.373232 18256 heartbeater.cc:347] Connected to a master server at 127.0.196.254:35921
I20260925 19:08:45.373333 18256 heartbeater.cc:464] Registering TS with master...
I20260925 19:08:45.373509 18256 heartbeater.cc:511] Master 127.0.196.254:35921 requested a full tablet report, sending...
I20260925 19:08:45.373926 17797 ts_manager.cc:194] Registered new tserver with Master: 898332ff3e624b0c8e8482382ee89857 (127.0.196.195:35303)
I20260925 19:08:45.374322 17797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.195@KRBTEST.COM'} at 127.0.196.195:40225
I20260925 19:08:45.377487 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:08:45.384972 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:45.378961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45950 (local address 127.0.196.254:35921)
0925 19:08:45.379020 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:45.379022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:45.379113 (+ 91us) server_negotiation.cc:407] Connection header received
0925 19:08:45.379201 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:45.379202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:45.379212 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:45.379276 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:45.379841 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.380335 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.381120 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.381321 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.381432 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:45.382613 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:45.382637 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:45.382638 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:08:45.382646 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:45.383799 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:45.384159 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:45.384161 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:45.384161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:45.384218 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:45.384413 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:45.384414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:45.384414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:45.384569 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:45.384674 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:45.384782 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 19:08:45.384869 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 19:08:45.386467 17797 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:08:45.388106 17797 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:08:45.394868 18272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:45.390679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60576 (local address 127.0.196.193:45393)
0925 19:08:45.390961 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:45.390965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:45.390979 (+ 14us) server_negotiation.cc:407] Connection header received
0925 19:08:45.391181 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:45.391184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:45.391200 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:45.391275 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:45.391536 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.392229 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.393836 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.394351 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.394441 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:45.394510 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:45.394724 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 19:08:45.394782 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":90,"threads_started":1}
I20260925 19:08:45.395058 18271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:45.390670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51278 (local address 127.0.196.194:45379)
0925 19:08:45.390863 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:45.390868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:45.390881 (+ 13us) server_negotiation.cc:407] Connection header received
0925 19:08:45.390925 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:45.390928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:45.390944 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:45.391035 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:45.391249 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.392442 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.393788 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.394484 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.394542 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:45.394593 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:45.394912 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 19:08:45.394969 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":96,"threads_started":1}
I20260925 19:08:45.395058 18270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:45.390535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47044 (local address 127.0.196.195:35303)
0925 19:08:45.390902 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:45.390908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:45.391218 (+ 310us) server_negotiation.cc:407] Connection header received
0925 19:08:45.391263 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:45.391265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:45.391285 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:45.391358 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:45.391581 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.392601 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.393714 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.394458 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.394536 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:45.394593 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:45.394928 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 19:08:45.394980 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":119,"threads_started":1}
I20260925 19:08:45.394876 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:45.390386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:45393 (local address 127.0.0.1:60576)
0925 19:08:45.390929 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:45.390939 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:45.390993 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:45.391367 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:45.391368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:45.391370 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:45.391457 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:45.391458 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.392335 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.392337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:45.393711 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:45.393714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.394448 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.394449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:45.394551 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:45.394567 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 19:08:45.394681 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 19:08:45.394758 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":43,"thread_start_us":65,"threads_started":1}
I20260925 19:08:45.394876 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:45.390386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:45379 (local address 127.0.0.1:51278)
0925 19:08:45.390596 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:45.390674 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:45.390760 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:45.391053 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:45.391057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:45.391060 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:45.391159 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:45.391161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.392548 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.392548 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:45.393711 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:45.393714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.394560 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.394560 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:45.394618 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:45.394622 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:08:45.394706 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 19:08:45.394758 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":16,"thread_start_us":77,"threads_started":1}
I20260925 19:08:45.395248 18267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:45.390422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:35303 (local address 127.0.0.1:47044)
0925 19:08:45.391109 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:45.391119 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:45.391159 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:45.391447 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:45.391448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:45.391450 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:45.391499 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:45.391500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.392708 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.392709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:45.393617 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:45.393620 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.394658 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.394659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:45.394725 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:45.394730 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:08:45.394782 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 19:08:45.394824 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20260925 19:08:45.396239 18190 tablet_service.cc:1467] Processing CreateTablet for tablet e25f5ac6c3494f52b0fe52e8a34a6cd1 (DEFAULT_TABLE table=test-table [id=939e2e4ed2e448998042eb10934ee1b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:08:45.396256 18056 tablet_service.cc:1467] Processing CreateTablet for tablet e25f5ac6c3494f52b0fe52e8a34a6cd1 (DEFAULT_TABLE table=test-table [id=939e2e4ed2e448998042eb10934ee1b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:08:45.396260 17922 tablet_service.cc:1467] Processing CreateTablet for tablet e25f5ac6c3494f52b0fe52e8a34a6cd1 (DEFAULT_TABLE table=test-table [id=939e2e4ed2e448998042eb10934ee1b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:08:45.396605 18190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e25f5ac6c3494f52b0fe52e8a34a6cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:45.396605 18056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e25f5ac6c3494f52b0fe52e8a34a6cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:45.396605 17922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e25f5ac6c3494f52b0fe52e8a34a6cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:45.399451 18273 tablet_bootstrap.cc:492] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P 898332ff3e624b0c8e8482382ee89857: Bootstrap starting.
I20260925 19:08:45.400076 18275 tablet_bootstrap.cc:492] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P ac4ee18cd64440b39bec2381b4e565b8: Bootstrap starting.
I20260925 19:08:45.400306 18273 tablet_bootstrap.cc:654] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P 898332ff3e624b0c8e8482382ee89857: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:45.400774 18275 tablet_bootstrap.cc:654] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P ac4ee18cd64440b39bec2381b4e565b8: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:45.401033 18273 log.cc:824] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P 898332ff3e624b0c8e8482382ee89857: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:45.401347 18275 log.cc:824] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P ac4ee18cd64440b39bec2381b4e565b8: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:45.401682 18274 tablet_bootstrap.cc:492] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P a5fb57d4c98b4126a9b517a5da9c1b7a: Bootstrap starting.
I20260925 19:08:45.401948 18273 tablet_bootstrap.cc:492] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P 898332ff3e624b0c8e8482382ee89857: No bootstrap required, opened a new log
I20260925 19:08:45.402031 18273 ts_tablet_manager.cc:1397] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P 898332ff3e624b0c8e8482382ee89857: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 19:08:45.402076 18275 tablet_bootstrap.cc:492] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P ac4ee18cd64440b39bec2381b4e565b8: No bootstrap required, opened a new log
I20260925 19:08:45.402149 18275 ts_tablet_manager.cc:1397] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P ac4ee18cd64440b39bec2381b4e565b8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:08:45.402329 18274 tablet_bootstrap.cc:654] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P a5fb57d4c98b4126a9b517a5da9c1b7a: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:45.403056 18274 log.cc:824] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P a5fb57d4c98b4126a9b517a5da9c1b7a: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:45.403923 18274 tablet_bootstrap.cc:492] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P a5fb57d4c98b4126a9b517a5da9c1b7a: No bootstrap required, opened a new log
I20260925 19:08:45.404007 18274 ts_tablet_manager.cc:1397] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P a5fb57d4c98b4126a9b517a5da9c1b7a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 19:08:45.404207 18275 raft_consensus.cc:348] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P ac4ee18cd64440b39bec2381b4e565b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fb57d4c98b4126a9b517a5da9c1b7a" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45379 } } peers { permanent_uuid: "ac4ee18cd64440b39bec2381b4e565b8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45393 } } peers { permanent_uuid: "898332ff3e624b0c8e8482382ee89857" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35303 } }
I20260925 19:08:45.404377 18275 raft_consensus.cc:374] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P ac4ee18cd64440b39bec2381b4e565b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:45.404423 18275 raft_consensus.cc:729] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P ac4ee18cd64440b39bec2381b4e565b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac4ee18cd64440b39bec2381b4e565b8, State: Initialized, Role: FOLLOWER
I20260925 19:08:45.404295 18273 raft_consensus.cc:348] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P 898332ff3e624b0c8e8482382ee89857 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fb57d4c98b4126a9b517a5da9c1b7a" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45379 } } peers { permanent_uuid: "ac4ee18cd64440b39bec2381b4e565b8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45393 } } peers { permanent_uuid: "898332ff3e624b0c8e8482382ee89857" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35303 } }
I20260925 19:08:45.404476 18273 raft_consensus.cc:374] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P 898332ff3e624b0c8e8482382ee89857 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:45.404537 18273 raft_consensus.cc:729] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P 898332ff3e624b0c8e8482382ee89857 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 898332ff3e624b0c8e8482382ee89857, State: Initialized, Role: FOLLOWER
I20260925 19:08:45.404567 18275 consensus_queue.cc:260] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P ac4ee18cd64440b39bec2381b4e565b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fb57d4c98b4126a9b517a5da9c1b7a" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45379 } } peers { permanent_uuid: "ac4ee18cd64440b39bec2381b4e565b8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45393 } } peers { permanent_uuid: "898332ff3e624b0c8e8482382ee89857" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35303 } }
I20260925 19:08:45.404670 18273 consensus_queue.cc:260] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P 898332ff3e624b0c8e8482382ee89857 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fb57d4c98b4126a9b517a5da9c1b7a" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45379 } } peers { permanent_uuid: "ac4ee18cd64440b39bec2381b4e565b8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45393 } } peers { permanent_uuid: "898332ff3e624b0c8e8482382ee89857" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35303 } }
I20260925 19:08:45.404906 18275 ts_tablet_manager.cc:1428] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P ac4ee18cd64440b39bec2381b4e565b8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:08:45.404963 18256 heartbeater.cc:503] Master 127.0.196.254:35921 was elected leader, sending a full tablet report...
I20260925 19:08:45.404945 18273 ts_tablet_manager.cc:1428] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P 898332ff3e624b0c8e8482382ee89857: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:08:45.405030 17988 heartbeater.cc:503] Master 127.0.196.254:35921 was elected leader, sending a full tablet report...
I20260925 19:08:45.406082 18274 raft_consensus.cc:348] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P a5fb57d4c98b4126a9b517a5da9c1b7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fb57d4c98b4126a9b517a5da9c1b7a" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45379 } } peers { permanent_uuid: "ac4ee18cd64440b39bec2381b4e565b8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45393 } } peers { permanent_uuid: "898332ff3e624b0c8e8482382ee89857" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35303 } }
I20260925 19:08:45.406283 18274 raft_consensus.cc:374] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P a5fb57d4c98b4126a9b517a5da9c1b7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:45.406334 18274 raft_consensus.cc:729] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P a5fb57d4c98b4126a9b517a5da9c1b7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5fb57d4c98b4126a9b517a5da9c1b7a, State: Initialized, Role: FOLLOWER
I20260925 19:08:45.406474 18274 consensus_queue.cc:260] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P a5fb57d4c98b4126a9b517a5da9c1b7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fb57d4c98b4126a9b517a5da9c1b7a" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45379 } } peers { permanent_uuid: "ac4ee18cd64440b39bec2381b4e565b8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45393 } } peers { permanent_uuid: "898332ff3e624b0c8e8482382ee89857" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35303 } }
I20260925 19:08:45.406829 18122 heartbeater.cc:503] Master 127.0.196.254:35921 was elected leader, sending a full tablet report...
I20260925 19:08:45.406831 18274 ts_tablet_manager.cc:1428] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P a5fb57d4c98b4126a9b517a5da9c1b7a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260925 19:08:45.442245 18123 tablet.cc:2369] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P a5fb57d4c98b4126a9b517a5da9c1b7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:08:45.532188 17989 tablet.cc:2369] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P ac4ee18cd64440b39bec2381b4e565b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:08:45.612856 18257 tablet.cc:2369] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P 898332ff3e624b0c8e8482382ee89857: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:08:45.656491 18279 raft_consensus.cc:482] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P ac4ee18cd64440b39bec2381b4e565b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:08:45.656638 18279 raft_consensus.cc:504] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P ac4ee18cd64440b39bec2381b4e565b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fb57d4c98b4126a9b517a5da9c1b7a" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45379 } } peers { permanent_uuid: "ac4ee18cd64440b39bec2381b4e565b8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45393 } } peers { permanent_uuid: "898332ff3e624b0c8e8482382ee89857" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35303 } }
I20260925 19:08:45.657109 18279 leader_election.cc:288] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P ac4ee18cd64440b39bec2381b4e565b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5fb57d4c98b4126a9b517a5da9c1b7a (127.0.196.194:45379), 898332ff3e624b0c8e8482382ee89857 (127.0.196.195:35303)
I20260925 19:08:45.660255 18270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:45.657534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:50315 (local address 127.0.196.195:35303)
0925 19:08:45.657579 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:45.657581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:45.657608 (+ 27us) server_negotiation.cc:407] Connection header received
0925 19:08:45.657637 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:45.657638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:45.657646 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:45.657716 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:45.657888 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.658366 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.659477 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.659870 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.659970 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:45.659979 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:45.660147 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 19:08:45.660185 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:08:45.660267 18271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:45.657437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:38561 (local address 127.0.196.194:45379)
0925 19:08:45.657506 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:45.657509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:45.657550 (+ 41us) server_negotiation.cc:407] Connection header received
0925 19:08:45.657619 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:45.657620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:45.657630 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:45.657701 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:45.657859 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.658343 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.659477 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.659871 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.659971 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:45.659981 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:45.660155 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 19:08:45.660196 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 19:08:45.660301 18282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:45.657337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:45379 (local address 127.0.196.193:38561)
0925 19:08:45.657469 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:45.657482 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:45.657556 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:45.657706 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:45.657707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:45.657709 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:45.657787 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:45.657789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.658453 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.658454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:45.659375 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:45.659378 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.659981 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.659982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:45.660061 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:45.660064 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:08:45.660146 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 19:08:45.660189 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":72,"thread_start_us":62,"threads_started":1}
I20260925 19:08:45.660323 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:45.657337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:35303 (local address 127.0.196.193:50315)
0925 19:08:45.657483 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:45.657496 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:45.657572 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:45.657722 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:45.657723 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:45.657725 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:45.657806 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:45.657807 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.658456 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.658457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:45.659376 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:45.659379 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.659970 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.659971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:45.660051 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:45.660055 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:08:45.660122 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 19:08:45.660172 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":21,"thread_start_us":64,"threads_started":1}
I20260925 19:08:45.660686 18076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25f5ac6c3494f52b0fe52e8a34a6cd1" candidate_uuid: "ac4ee18cd64440b39bec2381b4e565b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5fb57d4c98b4126a9b517a5da9c1b7a" is_pre_election: true
I20260925 19:08:45.660876 18076 raft_consensus.cc:2454] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P a5fb57d4c98b4126a9b517a5da9c1b7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac4ee18cd64440b39bec2381b4e565b8 in term 0.
I20260925 19:08:45.660862 18210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25f5ac6c3494f52b0fe52e8a34a6cd1" candidate_uuid: "ac4ee18cd64440b39bec2381b4e565b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "898332ff3e624b0c8e8482382ee89857" is_pre_election: true
I20260925 19:08:45.661033 18210 raft_consensus.cc:2454] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P 898332ff3e624b0c8e8482382ee89857 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac4ee18cd64440b39bec2381b4e565b8 in term 0.
I20260925 19:08:45.661120 17877 leader_election.cc:302] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P ac4ee18cd64440b39bec2381b4e565b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5fb57d4c98b4126a9b517a5da9c1b7a, ac4ee18cd64440b39bec2381b4e565b8; no voters:
I20260925 19:08:45.661259 18279 raft_consensus.cc:2781] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P ac4ee18cd64440b39bec2381b4e565b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:08:45.661324 18279 raft_consensus.cc:482] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P ac4ee18cd64440b39bec2381b4e565b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:08:45.661355 18279 raft_consensus.cc:3037] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P ac4ee18cd64440b39bec2381b4e565b8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:45.662070 18279 raft_consensus.cc:504] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P ac4ee18cd64440b39bec2381b4e565b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fb57d4c98b4126a9b517a5da9c1b7a" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45379 } } peers { permanent_uuid: "ac4ee18cd64440b39bec2381b4e565b8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45393 } } peers { permanent_uuid: "898332ff3e624b0c8e8482382ee89857" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35303 } }
I20260925 19:08:45.662233 18279 leader_election.cc:288] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P ac4ee18cd64440b39bec2381b4e565b8 [CANDIDATE]: Term 1 election: Requested vote from peers a5fb57d4c98b4126a9b517a5da9c1b7a (127.0.196.194:45379), 898332ff3e624b0c8e8482382ee89857 (127.0.196.195:35303)
I20260925 19:08:45.662384 18210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25f5ac6c3494f52b0fe52e8a34a6cd1" candidate_uuid: "ac4ee18cd64440b39bec2381b4e565b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "898332ff3e624b0c8e8482382ee89857"
I20260925 19:08:45.662472 18210 raft_consensus.cc:3037] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P 898332ff3e624b0c8e8482382ee89857 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:45.662500 18076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25f5ac6c3494f52b0fe52e8a34a6cd1" candidate_uuid: "ac4ee18cd64440b39bec2381b4e565b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5fb57d4c98b4126a9b517a5da9c1b7a"
I20260925 19:08:45.662593 18076 raft_consensus.cc:3037] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P a5fb57d4c98b4126a9b517a5da9c1b7a [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:45.663264 18210 raft_consensus.cc:2454] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P 898332ff3e624b0c8e8482382ee89857 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac4ee18cd64440b39bec2381b4e565b8 in term 1.
I20260925 19:08:45.663345 18076 raft_consensus.cc:2454] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P a5fb57d4c98b4126a9b517a5da9c1b7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac4ee18cd64440b39bec2381b4e565b8 in term 1.
I20260925 19:08:45.663457 17876 leader_election.cc:302] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P ac4ee18cd64440b39bec2381b4e565b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 898332ff3e624b0c8e8482382ee89857, ac4ee18cd64440b39bec2381b4e565b8; no voters:
I20260925 19:08:45.663556 18279 raft_consensus.cc:2781] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P ac4ee18cd64440b39bec2381b4e565b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:08:45.663707 18279 raft_consensus.cc:686] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P ac4ee18cd64440b39bec2381b4e565b8 [term 1 LEADER]: Becoming Leader. State: Replica: ac4ee18cd64440b39bec2381b4e565b8, State: Running, Role: LEADER
I20260925 19:08:45.663805 18279 consensus_queue.cc:237] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P ac4ee18cd64440b39bec2381b4e565b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fb57d4c98b4126a9b517a5da9c1b7a" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45379 } } peers { permanent_uuid: "ac4ee18cd64440b39bec2381b4e565b8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45393 } } peers { permanent_uuid: "898332ff3e624b0c8e8482382ee89857" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35303 } }
I20260925 19:08:45.664709 17797 catalog_manager.cc:5495] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P ac4ee18cd64440b39bec2381b4e565b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac4ee18cd64440b39bec2381b4e565b8 (127.0.196.193). New cstate: current_term: 1 leader_uuid: "ac4ee18cd64440b39bec2381b4e565b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fb57d4c98b4126a9b517a5da9c1b7a" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac4ee18cd64440b39bec2381b4e565b8" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 45393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "898332ff3e624b0c8e8482382ee89857" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35303 } health_report { overall_health: UNKNOWN } } }
I20260925 19:08:45.682309 18272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:45.680137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60588 (local address 127.0.196.193:45393)
0925 19:08:45.680197 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:45.680199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:45.680208 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:08:45.680253 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:45.680254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:45.680272 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:45.680336 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:08:45.680469 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.680935 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.681649 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.681829 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.681877 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:45.682026 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:08:45.682135 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:08:45.682198 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:45.682227 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 19:08:45.682243 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:08:45.684280 18210 raft_consensus.cc:1261] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P 898332ff3e624b0c8e8482382ee89857 [term 1 FOLLOWER]: Refusing update from remote peer ac4ee18cd64440b39bec2381b4e565b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:08:45.684285 18076 raft_consensus.cc:1261] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P a5fb57d4c98b4126a9b517a5da9c1b7a [term 1 FOLLOWER]: Refusing update from remote peer ac4ee18cd64440b39bec2381b4e565b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:08:45.684659 18279 consensus_queue.cc:1035] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P ac4ee18cd64440b39bec2381b4e565b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5fb57d4c98b4126a9b517a5da9c1b7a" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 45379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:08:45.684800 18284 consensus_queue.cc:1035] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P ac4ee18cd64440b39bec2381b4e565b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "898332ff3e624b0c8e8482382ee89857" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:08:45.687817 18289 mvcc.cc:205] Tried to move back new op lower bound from 7333328181999771648 to 7333328181919596544. Current Snapshot: MvccSnapshot[applied={T|T < 7333328181999771648 or (T in {7333328181999771648})}]
I20260925 19:08:45.688341 18288 mvcc.cc:205] Tried to move back new op lower bound from 7333328181999771648 to 7333328181919596544. Current Snapshot: MvccSnapshot[applied={T|T < 7333328181999771648 or (T in {7333328181999771648})}]
I20260925 19:08:45.689741 17797 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:08:45.689829 17797 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:08:45.690523 17797 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4677ec3b29e44c2862628476f21165a: Sending DeleteTablet for 3 replicas of tablet e25f5ac6c3494f52b0fe52e8a34a6cd1
I20260925 19:08:45.690923 18056 tablet_service.cc:1514] Processing DeleteTablet for tablet e25f5ac6c3494f52b0fe52e8a34a6cd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:08:45 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:51278
I20260925 19:08:45.690981 18190 tablet_service.cc:1514] Processing DeleteTablet for tablet e25f5ac6c3494f52b0fe52e8a34a6cd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:08:45 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:47044
I20260925 19:08:45.691197 17922 tablet_service.cc:1514] Processing DeleteTablet for tablet e25f5ac6c3494f52b0fe52e8a34a6cd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:08:45 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:60576
I20260925 19:08:45.691296 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 17859
I20260925 19:08:45.691835 18296 tablet_replica.cc:330] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P a5fb57d4c98b4126a9b517a5da9c1b7a: stopping tablet replica
I20260925 19:08:45.693235 18297 tablet_replica.cc:330] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P 898332ff3e624b0c8e8482382ee89857: stopping tablet replica
I20260925 19:08:45.693244 18296 raft_consensus.cc:2229] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P a5fb57d4c98b4126a9b517a5da9c1b7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:08:45.693351 18297 raft_consensus.cc:2229] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P 898332ff3e624b0c8e8482382ee89857 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:08:45.693378 18296 raft_consensus.cc:2258] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P a5fb57d4c98b4126a9b517a5da9c1b7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:08:45.693464 18297 raft_consensus.cc:2258] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P 898332ff3e624b0c8e8482382ee89857 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:08:45.693854 18296 ts_tablet_manager.cc:1905] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P a5fb57d4c98b4126a9b517a5da9c1b7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:08:45.693900 18297 ts_tablet_manager.cc:1905] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P 898332ff3e624b0c8e8482382ee89857: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:08:45.695297 18296 ts_tablet_manager.cc:1918] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P a5fb57d4c98b4126a9b517a5da9c1b7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:08:45.695387 18296 log.cc:1196] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P a5fb57d4c98b4126a9b517a5da9c1b7a: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-1/wal/wals/e25f5ac6c3494f52b0fe52e8a34a6cd1
I20260925 19:08:45.695641 18296 ts_tablet_manager.cc:1939] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P a5fb57d4c98b4126a9b517a5da9c1b7a: Deleting consensus metadata
I20260925 19:08:45.695926 18297 ts_tablet_manager.cc:1918] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P 898332ff3e624b0c8e8482382ee89857: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:08:45.696099 18297 log.cc:1196] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P 898332ff3e624b0c8e8482382ee89857: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790363087086945-787-0/minicluster-data/ts-2/wal/wals/e25f5ac6c3494f52b0fe52e8a34a6cd1
I20260925 19:08:45.696115 17783 catalog_manager.cc:4841] TS a5fb57d4c98b4126a9b517a5da9c1b7a (127.0.196.194:45379): tablet e25f5ac6c3494f52b0fe52e8a34a6cd1 (table test-table [id=939e2e4ed2e448998042eb10934ee1b5]) successfully deleted
I20260925 19:08:45.696382 18297 ts_tablet_manager.cc:1939] T e25f5ac6c3494f52b0fe52e8a34a6cd1 P 898332ff3e624b0c8e8482382ee89857: Deleting consensus metadata
I20260925 19:08:45.696774 17782 catalog_manager.cc:4841] TS 898332ff3e624b0c8e8482382ee89857 (127.0.196.195:35303): tablet e25f5ac6c3494f52b0fe52e8a34a6cd1 (table test-table [id=939e2e4ed2e448998042eb10934ee1b5]) successfully deleted
W20260925 19:08:45.699002 17784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.196.193:45393 (error 108)
I20260925 19:08:45.699249 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 17993
I20260925 19:08:45.699870 18267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:45.699404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:45393 (local address 127.0.0.1:60592)
0925 19:08:45.699794 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:45.699838 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.193:45393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371}
W20260925 19:08:45.700048 17784 catalog_manager.cc:4584] TS ac4ee18cd64440b39bec2381b4e565b8 (127.0.196.193:45393): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e25f5ac6c3494f52b0fe52e8a34a6cd1: Network error: Client connection negotiation failed: client connection to 127.0.196.193:45393: connect: Connection refused (error 111)
I20260925 19:08:45.704687 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 18127
I20260925 19:08:45.709440 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 17765
2026-09-25T19:08:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2187 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T19:08:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:08:45Z Disabled control of system clock
I20260925 19:08:45.739441 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:44059
--webserver_interface=127.0.196.254
--webserver_port=0
--builtin_ntp_servers=127.0.196.212:36731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:44059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 19:08:45.824572 18303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:45.824817 18303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:45.824843 18303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:45.826042 18303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:08:45.826086 18303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:45.826109 18303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:45.826123 18303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:08:45.826136 18303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:08:45.827684 18303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:36731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:44059
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:44059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:45.827940 18303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:45.828212 18303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:08:45.833572 18303 server_base.cc:1034] running on GCE node
W20260925 19:08:45.833544 18311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:45.833534 18308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:45.833527 18309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:45.833999 18303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:45.834326 18303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:45.835495 18303 hybrid_clock.cc:648] HybridClock initialized: now 1790363325835482 us; error 45 us; skew 500 ppm
I20260925 19:08:45.836975 18303 webserver.cc:460] Webserver started at http://127.0.196.254:44799/ using document root <none> and password file <none>
I20260925 19:08:45.837253 18303 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:45.837325 18303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:45.837474 18303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:45.838668 18303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "4e5b7766240e4a7087104485e35ead78"
format_stamp: "Formatted at 2026-09-25 19:08:45 on dist-test-slave-rn98"
server_key: "72451d6f452fa4d525046fcb429d3a45"
server_key_iv: "140f9f8bfaddc6a0ceb691432e27aa2e"
server_key_version: "encryptionkey@0"
I20260925 19:08:45.839067 18303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "4e5b7766240e4a7087104485e35ead78"
format_stamp: "Formatted at 2026-09-25 19:08:45 on dist-test-slave-rn98"
server_key: "72451d6f452fa4d525046fcb429d3a45"
server_key_iv: "140f9f8bfaddc6a0ceb691432e27aa2e"
server_key_version: "encryptionkey@0"
I20260925 19:08:45.840538 18303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 19:08:45.841424 18317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:45.841599 18303 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:08:45.841713 18303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "4e5b7766240e4a7087104485e35ead78"
format_stamp: "Formatted at 2026-09-25 19:08:45 on dist-test-slave-rn98"
server_key: "72451d6f452fa4d525046fcb429d3a45"
server_key_iv: "140f9f8bfaddc6a0ceb691432e27aa2e"
server_key_version: "encryptionkey@0"
I20260925 19:08:45.841794 18303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:45.869774 18303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:45.870114 18303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:45.870273 18303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:45.875054 18303 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:44059
I20260925 19:08:45.875124 18369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:44059 every 8 connection(s)
I20260925 19:08:45.875506 18303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:08:45.876597 18370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:45.879281 18370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e5b7766240e4a7087104485e35ead78: Bootstrap starting.
I20260925 19:08:45.879909 18370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e5b7766240e4a7087104485e35ead78: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:45.880218 18370 log.cc:824] T 00000000000000000000000000000000 P 4e5b7766240e4a7087104485e35ead78: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:45.880992 18370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e5b7766240e4a7087104485e35ead78: No bootstrap required, opened a new log
I20260925 19:08:45.882659 18370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e5b7766240e4a7087104485e35ead78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e5b7766240e4a7087104485e35ead78" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 44059 } }
I20260925 19:08:45.882822 18370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e5b7766240e4a7087104485e35ead78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:45.882845 18370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e5b7766240e4a7087104485e35ead78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e5b7766240e4a7087104485e35ead78, State: Initialized, Role: FOLLOWER
I20260925 19:08:45.883008 18370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e5b7766240e4a7087104485e35ead78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e5b7766240e4a7087104485e35ead78" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 44059 } }
I20260925 19:08:45.883082 18370 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e5b7766240e4a7087104485e35ead78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:08:45.883106 18370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e5b7766240e4a7087104485e35ead78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:08:45.883172 18370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e5b7766240e4a7087104485e35ead78 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:45.883749 18370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e5b7766240e4a7087104485e35ead78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e5b7766240e4a7087104485e35ead78" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 44059 } }
I20260925 19:08:45.883862 18370 leader_election.cc:302] T 00000000000000000000000000000000 P 4e5b7766240e4a7087104485e35ead78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e5b7766240e4a7087104485e35ead78; no voters:
I20260925 19:08:45.884060 18370 leader_election.cc:288] T 00000000000000000000000000000000 P 4e5b7766240e4a7087104485e35ead78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:08:45.884152 18373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e5b7766240e4a7087104485e35ead78 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:08:45.884325 18373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e5b7766240e4a7087104485e35ead78 [term 1 LEADER]: Becoming Leader. State: Replica: 4e5b7766240e4a7087104485e35ead78, State: Running, Role: LEADER
I20260925 19:08:45.884351 18370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e5b7766240e4a7087104485e35ead78 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:08:45.884459 18373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e5b7766240e4a7087104485e35ead78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e5b7766240e4a7087104485e35ead78" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 44059 } }
I20260925 19:08:45.884598 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 18303
I20260925 19:08:45.884699 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:08:45.884938 787 external_mini_cluster.cc:1428] Setting key 586f37456f058eff0f2e45e168b7106f
I20260925 19:08:45.885041 18374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e5b7766240e4a7087104485e35ead78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e5b7766240e4a7087104485e35ead78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e5b7766240e4a7087104485e35ead78" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 44059 } } }
I20260925 19:08:45.885130 18374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e5b7766240e4a7087104485e35ead78 [sys.catalog]: This master's current role is: LEADER
I20260925 19:08:45.885366 18382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:08:45.885933 18382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:08:45.885988 18375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e5b7766240e4a7087104485e35ead78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e5b7766240e4a7087104485e35ead78. Latest consensus state: current_term: 1 leader_uuid: "4e5b7766240e4a7087104485e35ead78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e5b7766240e4a7087104485e35ead78" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 44059 } } }
I20260925 19:08:45.886117 18375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e5b7766240e4a7087104485e35ead78 [sys.catalog]: This master's current role is: LEADER
I20260925 19:08:45.888135 18382 catalog_manager.cc:1347] Generated new cluster ID: a0d09584fe1c46d698c1ecc56ce90c5e
I20260925 19:08:45.888203 18382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:08:45.891397 18390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:45.885661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43666 (local address 127.0.196.254:44059)
0925 19:08:45.886145 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:45.886155 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:45.886201 (+ 46us) server_negotiation.cc:407] Connection header received
0925 19:08:45.886374 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:45.886382 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:45.886459 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:45.886650 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:45.886848 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.888019 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.889108 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:45.889728 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:45.889856 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:45.890577 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:45.890582 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:45.890582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:08:45.890843 (+ 261us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:45.890906 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:08:45.890929 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:45.890994 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:45.891097 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 19:08:45.891179 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":60,"threads_started":1}
I20260925 19:08:45.902725 18382 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:08:45.903343 18382 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:08:45.905592 18382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e5b7766240e4a7087104485e35ead78: Generated new TSK 0
I20260925 19:08:45.905805 18382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:08:45.912813 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:0
--local_ip_for_outbound_sockets=127.0.196.193
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:44059
--builtin_ntp_servers=127.0.196.212:36731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 19:08:45.998206 18394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:45.998428 18394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:45.998453 18394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:45.999781 18394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:45.999835 18394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:45.999863 18394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:08:46.001446 18394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:36731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:44059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:46.001736 18394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:46.002069 18394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:46.007318 18399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:46.007376 18402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:46.007477 18394 server_base.cc:1034] running on GCE node
W20260925 19:08:46.007576 18400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:46.007830 18394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:46.008177 18394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:46.009384 18394 hybrid_clock.cc:648] HybridClock initialized: now 1790363326009362 us; error 46 us; skew 500 ppm
I20260925 19:08:46.010916 18394 webserver.cc:460] Webserver started at http://127.0.196.193:40001/ using document root <none> and password file <none>
I20260925 19:08:46.011205 18394 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:46.011264 18394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:46.011402 18394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:46.012876 18394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-0/data/instance:
uuid: "fb75ec7597464facab4bd3c542141f69"
format_stamp: "Formatted at 2026-09-25 19:08:46 on dist-test-slave-rn98"
server_key: "c20478e8bf92bb5a0360a550ecb58670"
server_key_iv: "b09db5431dc4fd289cada28e65745bc0"
server_key_version: "encryptionkey@0"
I20260925 19:08:46.013303 18394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-0/wal/instance:
uuid: "fb75ec7597464facab4bd3c542141f69"
format_stamp: "Formatted at 2026-09-25 19:08:46 on dist-test-slave-rn98"
server_key: "c20478e8bf92bb5a0360a550ecb58670"
server_key_iv: "b09db5431dc4fd289cada28e65745bc0"
server_key_version: "encryptionkey@0"
I20260925 19:08:46.014999 18394 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260925 19:08:46.015974 18408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:46.016214 18394 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:08:46.016314 18394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "fb75ec7597464facab4bd3c542141f69"
format_stamp: "Formatted at 2026-09-25 19:08:46 on dist-test-slave-rn98"
server_key: "c20478e8bf92bb5a0360a550ecb58670"
server_key_iv: "b09db5431dc4fd289cada28e65745bc0"
server_key_version: "encryptionkey@0"
I20260925 19:08:46.016397 18394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:46.065006 18394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:46.065315 18394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:46.065426 18394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:46.065631 18394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:08:46.066092 18394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:08:46.066131 18394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:46.066156 18394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:08:46.066207 18394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:46.072585 18394 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:41997
I20260925 19:08:46.072630 18521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:41997 every 8 connection(s)
I20260925 19:08:46.073024 18394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
I20260925 19:08:46.078569 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 18394
I20260925 19:08:46.078788 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-0/wal/instance
I20260925 19:08:46.079071 787 external_mini_cluster.cc:1428] Setting key e82e52c295b89170294a8f7ac69fac5a
I20260925 19:08:46.079099 18390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:46.073920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:50819 (local address 127.0.196.254:44059)
0925 19:08:46.074003 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:46.074006 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:46.074191 (+ 185us) server_negotiation.cc:407] Connection header received
0925 19:08:46.074747 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:46.074749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:46.074770 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:46.074833 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:46.075586 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.076156 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.077316 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.077535 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.077608 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:46.078715 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:46.078718 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:46.078718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:08:46.078728 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:46.078747 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:08:46.078753 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:46.078816 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:46.078995 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 19:08:46.079036 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 19:08:46.079417 18524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:46.073840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:44059 (local address 127.0.196.193:50819)
0925 19:08:46.074049 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:46.074090 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:46.074656 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:46.074898 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:46.074902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:46.074912 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:08:46.075489 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:46.075494 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.076267 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.076268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:46.077227 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:46.077229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.077648 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.077649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:46.077810 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:46.078491 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:08:46.078492 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:08:46.078546 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:08:46.078583 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:08:46.078882 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:08:46.078892 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:08:46.078896 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:08:46.078999 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 19:08:46.079148 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":105,"threads_started":1}
I20260925 19:08:46.079954 18522 heartbeater.cc:347] Connected to a master server at 127.0.196.254:44059
I20260925 19:08:46.080073 18522 heartbeater.cc:464] Registering TS with master...
I20260925 19:08:46.080123 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:0
--local_ip_for_outbound_sockets=127.0.196.194
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:44059
--builtin_ntp_servers=127.0.196.212:36731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 19:08:46.080312 18522 heartbeater.cc:511] Master 127.0.196.254:44059 requested a full tablet report, sending...
I20260925 19:08:46.080825 18334 ts_manager.cc:194] Registered new tserver with Master: fb75ec7597464facab4bd3c542141f69 (127.0.196.193:41997)
I20260925 19:08:46.081737 18334 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.196.193:50819
W20260925 19:08:46.163357 18525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:46.163591 18525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:46.163616 18525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:46.164824 18525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:46.164870 18525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:46.164896 18525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:08:46.166955 18525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:36731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:44059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:46.167241 18525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:46.167567 18525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:08:46.173115 18525 server_base.cc:1034] running on GCE node
W20260925 19:08:46.173105 18533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:46.173106 18531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:46.173273 18530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:46.173545 18525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:46.173854 18525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:46.175040 18525 hybrid_clock.cc:648] HybridClock initialized: now 1790363326175026 us; error 47 us; skew 500 ppm
I20260925 19:08:46.176504 18525 webserver.cc:460] Webserver started at http://127.0.196.194:39879/ using document root <none> and password file <none>
I20260925 19:08:46.176784 18525 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:46.176860 18525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:46.177004 18525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:46.178128 18525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-1/data/instance:
uuid: "e4da7e62b8984ffc8e0f7faa579ef303"
format_stamp: "Formatted at 2026-09-25 19:08:46 on dist-test-slave-rn98"
server_key: "7a9d9291ee3f471ab3277885535b27d3"
server_key_iv: "7d776fa1a3b8f3de1355bc005ee47c1a"
server_key_version: "encryptionkey@0"
I20260925 19:08:46.178541 18525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-1/wal/instance:
uuid: "e4da7e62b8984ffc8e0f7faa579ef303"
format_stamp: "Formatted at 2026-09-25 19:08:46 on dist-test-slave-rn98"
server_key: "7a9d9291ee3f471ab3277885535b27d3"
server_key_iv: "7d776fa1a3b8f3de1355bc005ee47c1a"
server_key_version: "encryptionkey@0"
I20260925 19:08:46.179939 18525 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:08:46.180723 18539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:46.180920 18525 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:08:46.181025 18525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "e4da7e62b8984ffc8e0f7faa579ef303"
format_stamp: "Formatted at 2026-09-25 19:08:46 on dist-test-slave-rn98"
server_key: "7a9d9291ee3f471ab3277885535b27d3"
server_key_iv: "7d776fa1a3b8f3de1355bc005ee47c1a"
server_key_version: "encryptionkey@0"
I20260925 19:08:46.181111 18525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:46.206928 18525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:46.207283 18525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:46.207442 18525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:46.207683 18525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:08:46.208061 18525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:08:46.208117 18525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:46.208160 18525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:08:46.208189 18525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:46.214334 18525 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:35583
I20260925 19:08:46.214385 18652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:35583 every 8 connection(s)
I20260925 19:08:46.214829 18525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
I20260925 19:08:46.215723 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 18525
I20260925 19:08:46.215889 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-1/wal/instance
I20260925 19:08:46.216102 787 external_mini_cluster.cc:1428] Setting key 50b7b8bbc4156d30990d52af79710df9
I20260925 19:08:46.217023 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:0
--local_ip_for_outbound_sockets=127.0.196.195
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:44059
--builtin_ntp_servers=127.0.196.212:36731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 19:08:46.220572 18390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:46.215656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:56297 (local address 127.0.196.254:44059)
0925 19:08:46.215740 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:46.215743 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:46.215958 (+ 215us) server_negotiation.cc:407] Connection header received
0925 19:08:46.216617 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:46.216618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:46.216631 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:46.216682 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:46.217483 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.217981 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.219098 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.219297 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.219368 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:46.220261 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:46.220263 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:46.220263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:08:46.220269 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:46.220285 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:08:46.220290 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:46.220356 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:46.220451 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 19:08:46.220484 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 19:08:46.220834 18655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:46.215601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:44059 (local address 127.0.196.194:56297)
0925 19:08:46.215801 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:46.215846 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:46.216508 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:46.216787 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:46.216790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:46.216800 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:08:46.217379 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:46.217385 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.218083 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.218085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:46.218996 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:46.218998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.219389 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.219390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:46.219531 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:46.220094 (+ 563us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:08:46.220095 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:08:46.220128 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:08:46.220154 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:08:46.220355 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:08:46.220362 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:08:46.220366 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:08:46.220453 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 19:08:46.220626 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
I20260925 19:08:46.221338 18653 heartbeater.cc:347] Connected to a master server at 127.0.196.254:44059
I20260925 19:08:46.221449 18653 heartbeater.cc:464] Registering TS with master...
I20260925 19:08:46.221697 18653 heartbeater.cc:511] Master 127.0.196.254:44059 requested a full tablet report, sending...
I20260925 19:08:46.222112 18334 ts_manager.cc:194] Registered new tserver with Master: e4da7e62b8984ffc8e0f7faa579ef303 (127.0.196.194:35583)
I20260925 19:08:46.222718 18334 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.196.194:56297
W20260925 19:08:46.300737 18656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:46.300971 18656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:46.300994 18656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:46.302234 18656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:46.302281 18656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:46.302310 18656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:08:46.303771 18656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:36731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:44059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:46.304060 18656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:46.304337 18656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:46.309652 18664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:46.309659 18661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:46.309674 18662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:46.309680 18656 server_base.cc:1034] running on GCE node
I20260925 19:08:46.310079 18656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:46.310307 18656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:46.311463 18656 hybrid_clock.cc:648] HybridClock initialized: now 1790363326311451 us; error 33 us; skew 500 ppm
I20260925 19:08:46.312999 18656 webserver.cc:460] Webserver started at http://127.0.196.195:45583/ using document root <none> and password file <none>
I20260925 19:08:46.313323 18656 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:46.313405 18656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:46.313565 18656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:46.314894 18656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-2/data/instance:
uuid: "cd5b7e5c75c5474aa6f8a23956cbe01d"
format_stamp: "Formatted at 2026-09-25 19:08:46 on dist-test-slave-rn98"
server_key: "157562596b901bd68cbdde81eb905edc"
server_key_iv: "5dde7f37791a09b1f7091426d996bc5a"
server_key_version: "encryptionkey@0"
I20260925 19:08:46.315307 18656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-2/wal/instance:
uuid: "cd5b7e5c75c5474aa6f8a23956cbe01d"
format_stamp: "Formatted at 2026-09-25 19:08:46 on dist-test-slave-rn98"
server_key: "157562596b901bd68cbdde81eb905edc"
server_key_iv: "5dde7f37791a09b1f7091426d996bc5a"
server_key_version: "encryptionkey@0"
I20260925 19:08:46.316871 18656 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:08:46.317768 18670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:46.318003 18656 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:08:46.318105 18656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "cd5b7e5c75c5474aa6f8a23956cbe01d"
format_stamp: "Formatted at 2026-09-25 19:08:46 on dist-test-slave-rn98"
server_key: "157562596b901bd68cbdde81eb905edc"
server_key_iv: "5dde7f37791a09b1f7091426d996bc5a"
server_key_version: "encryptionkey@0"
I20260925 19:08:46.318178 18656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:46.325652 18656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:46.325996 18656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:46.326148 18656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:46.326390 18656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:08:46.326772 18656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:08:46.326831 18656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:46.326874 18656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:08:46.326901 18656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:46.332844 18656 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:45875
I20260925 19:08:46.332890 18783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:45875 every 8 connection(s)
I20260925 19:08:46.333242 18656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
I20260925 19:08:46.338789 18390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:46.334005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:49045 (local address 127.0.196.254:44059)
0925 19:08:46.334053 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:46.334055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:46.334318 (+ 263us) server_negotiation.cc:407] Connection header received
0925 19:08:46.334955 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:46.334956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:46.334970 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:46.335011 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:46.335733 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.336247 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.337283 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.337482 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.337555 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:46.338447 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:46.338449 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:46.338449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:08:46.338455 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:46.338476 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:08:46.338480 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:46.338536 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:46.338672 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 19:08:46.338704 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:08:46.339007 18786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:46.333982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:44059 (local address 127.0.196.195:49045)
0925 19:08:46.334192 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:46.334228 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:46.334833 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:46.335099 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:46.335103 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:46.335111 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:08:46.335633 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:46.335638 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.336353 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.336354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:46.337175 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:46.337177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.337565 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.337566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:46.337705 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:46.338276 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:08:46.338277 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:08:46.338309 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:08:46.338335 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:08:46.338549 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:08:46.338554 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:08:46.338558 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:08:46.338651 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 19:08:46.338763 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":91,"threads_started":1}
I20260925 19:08:46.339529 18784 heartbeater.cc:347] Connected to a master server at 127.0.196.254:44059
I20260925 19:08:46.339623 18784 heartbeater.cc:464] Registering TS with master...
I20260925 19:08:46.339780 18784 heartbeater.cc:511] Master 127.0.196.254:44059 requested a full tablet report, sending...
I20260925 19:08:46.340138 18334 ts_manager.cc:194] Registered new tserver with Master: cd5b7e5c75c5474aa6f8a23956cbe01d (127.0.196.195:45875)
I20260925 19:08:46.340471 18334 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.196.195:49045
I20260925 19:08:46.342355 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 18656
I20260925 19:08:46.342442 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-2/wal/instance
I20260925 19:08:46.342705 787 external_mini_cluster.cc:1428] Setting key 3f5f487341ba31fca697f4abc1ba74f6
I20260925 19:08:46.343608 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:08:46.347914 18390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:46.344962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43668 (local address 127.0.196.254:44059)
0925 19:08:46.345026 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:46.345028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:46.345191 (+ 163us) server_negotiation.cc:407] Connection header received
0925 19:08:46.345260 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:46.345261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:46.345289 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:46.345329 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:46.345472 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.345994 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.346838 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.347030 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.347087 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:46.347669 (+ 582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:46.347670 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:46.347670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:08:46.347675 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:46.347688 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:08:46.347691 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:46.347733 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:46.347773 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 19:08:46.347797 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 19:08:46.349558 18334 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:08:46.351255 18334 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:08:46.358048 18797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:46.353951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44422 (local address 127.0.196.193:41997)
0925 19:08:46.354181 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:46.354184 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:46.354194 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:08:46.354231 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:46.354233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:46.354247 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:46.354318 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:46.354673 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.355729 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.357114 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.357636 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.357726 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:46.357830 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:46.357909 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 19:08:46.357950 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":146,"threads_started":1}
I20260925 19:08:46.358035 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:46.353668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:41997 (local address 127.0.0.1:44422)
0925 19:08:46.353872 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:46.353882 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:46.353927 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:46.354319 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:46.354320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:46.354322 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:46.354372 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:46.354374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.355797 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.355798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:46.357017 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:46.357020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.357726 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.357727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:46.357801 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:46.357814 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 19:08:46.357902 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 19:08:46.357957 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":1,"thread_start_us":139,"threads_started":1}
I20260925 19:08:46.358326 18793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:46.353437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:45875 (local address 127.0.0.1:48948)
0925 19:08:46.353798 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:46.353821 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:46.353860 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:46.354314 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:46.354315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:46.354317 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:46.354353 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:46.354354 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.355705 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.355706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:46.357026 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:46.357035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.358093 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.358094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:46.358153 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:46.358156 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:08:46.358203 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 19:08:46.358244 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":39,"thread_start_us":89,"threads_started":1}
I20260925 19:08:46.358570 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:46.353544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48948 (local address 127.0.196.195:45875)
0925 19:08:46.353956 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:46.353961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:46.353973 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:08:46.354018 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:46.354020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:46.354042 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:46.354250 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:46.354421 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.355616 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.357133 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.357892 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.357966 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:46.358061 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:46.358441 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 19:08:46.358490 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":93,"threads_started":1}
I20260925 19:08:46.359329 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:46.353437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:35583 (local address 127.0.0.1:43620)
0925 19:08:46.353587 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:46.353635 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:46.353707 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:46.354215 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:46.354219 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:46.354222 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:46.354307 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:46.354310 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.356866 (+ 2556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.356867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:46.357945 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:46.357948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.359089 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.359091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:46.359155 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:46.359160 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:08:46.359214 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 19:08:46.359263 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260925 19:08:46.359366 18795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:46.353645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43620 (local address 127.0.196.194:35583)
0925 19:08:46.354034 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:46.354037 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:46.354045 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:08:46.354095 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:46.354097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:46.354115 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:46.354205 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:46.354467 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.356783 (+ 2316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.358039 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.359010 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.359077 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:46.359164 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:46.359225 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 19:08:46.359273 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":163,"threads_started":1}
I20260925 19:08:46.359232 18456 tablet_service.cc:1467] Processing CreateTablet for tablet e0f38a8973a44097a31dea1869c48b74 (DEFAULT_TABLE table=test-table [id=5213ee105e3d4e4b90e688f02aeead9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:08:46.359297 18718 tablet_service.cc:1467] Processing CreateTablet for tablet e0f38a8973a44097a31dea1869c48b74 (DEFAULT_TABLE table=test-table [id=5213ee105e3d4e4b90e688f02aeead9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:08:46.359654 18456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f38a8973a44097a31dea1869c48b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:46.359687 18718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f38a8973a44097a31dea1869c48b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:46.360203 18587 tablet_service.cc:1467] Processing CreateTablet for tablet e0f38a8973a44097a31dea1869c48b74 (DEFAULT_TABLE table=test-table [id=5213ee105e3d4e4b90e688f02aeead9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:08:46.360570 18587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f38a8973a44097a31dea1869c48b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:46.362591 18798 tablet_bootstrap.cc:492] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d: Bootstrap starting.
I20260925 19:08:46.363323 18798 tablet_bootstrap.cc:654] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:46.363291 18799 tablet_bootstrap.cc:492] T e0f38a8973a44097a31dea1869c48b74 P fb75ec7597464facab4bd3c542141f69: Bootstrap starting.
I20260925 19:08:46.363663 18798 log.cc:824] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:46.364020 18799 tablet_bootstrap.cc:654] T e0f38a8973a44097a31dea1869c48b74 P fb75ec7597464facab4bd3c542141f69: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:46.364333 18801 tablet_bootstrap.cc:492] T e0f38a8973a44097a31dea1869c48b74 P e4da7e62b8984ffc8e0f7faa579ef303: Bootstrap starting.
I20260925 19:08:46.364343 18799 log.cc:824] T e0f38a8973a44097a31dea1869c48b74 P fb75ec7597464facab4bd3c542141f69: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:46.364475 18798 tablet_bootstrap.cc:492] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d: No bootstrap required, opened a new log
I20260925 19:08:46.364552 18798 ts_tablet_manager.cc:1397] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 19:08:46.364939 18801 tablet_bootstrap.cc:654] T e0f38a8973a44097a31dea1869c48b74 P e4da7e62b8984ffc8e0f7faa579ef303: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:46.365406 18801 log.cc:824] T e0f38a8973a44097a31dea1869c48b74 P e4da7e62b8984ffc8e0f7faa579ef303: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:46.366185 18801 tablet_bootstrap.cc:492] T e0f38a8973a44097a31dea1869c48b74 P e4da7e62b8984ffc8e0f7faa579ef303: No bootstrap required, opened a new log
I20260925 19:08:46.366195 18799 tablet_bootstrap.cc:492] T e0f38a8973a44097a31dea1869c48b74 P fb75ec7597464facab4bd3c542141f69: No bootstrap required, opened a new log
I20260925 19:08:46.366267 18801 ts_tablet_manager.cc:1397] T e0f38a8973a44097a31dea1869c48b74 P e4da7e62b8984ffc8e0f7faa579ef303: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:08:46.366267 18799 ts_tablet_manager.cc:1397] T e0f38a8973a44097a31dea1869c48b74 P fb75ec7597464facab4bd3c542141f69: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 19:08:46.367003 18798 raft_consensus.cc:348] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4da7e62b8984ffc8e0f7faa579ef303" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 35583 } } peers { permanent_uuid: "cd5b7e5c75c5474aa6f8a23956cbe01d" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 45875 } } peers { permanent_uuid: "fb75ec7597464facab4bd3c542141f69" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 41997 } }
I20260925 19:08:46.367146 18798 raft_consensus.cc:374] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:46.367167 18798 raft_consensus.cc:729] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd5b7e5c75c5474aa6f8a23956cbe01d, State: Initialized, Role: FOLLOWER
I20260925 19:08:46.367291 18798 consensus_queue.cc:260] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4da7e62b8984ffc8e0f7faa579ef303" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 35583 } } peers { permanent_uuid: "cd5b7e5c75c5474aa6f8a23956cbe01d" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 45875 } } peers { permanent_uuid: "fb75ec7597464facab4bd3c542141f69" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 41997 } }
I20260925 19:08:46.367554 18798 ts_tablet_manager.cc:1428] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 19:08:46.367568 18784 heartbeater.cc:503] Master 127.0.196.254:44059 was elected leader, sending a full tablet report...
I20260925 19:08:46.369029 18801 raft_consensus.cc:348] T e0f38a8973a44097a31dea1869c48b74 P e4da7e62b8984ffc8e0f7faa579ef303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4da7e62b8984ffc8e0f7faa579ef303" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 35583 } } peers { permanent_uuid: "cd5b7e5c75c5474aa6f8a23956cbe01d" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 45875 } } peers { permanent_uuid: "fb75ec7597464facab4bd3c542141f69" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 41997 } }
I20260925 19:08:46.369029 18799 raft_consensus.cc:348] T e0f38a8973a44097a31dea1869c48b74 P fb75ec7597464facab4bd3c542141f69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4da7e62b8984ffc8e0f7faa579ef303" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 35583 } } peers { permanent_uuid: "cd5b7e5c75c5474aa6f8a23956cbe01d" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 45875 } } peers { permanent_uuid: "fb75ec7597464facab4bd3c542141f69" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 41997 } }
I20260925 19:08:46.369208 18799 raft_consensus.cc:374] T e0f38a8973a44097a31dea1869c48b74 P fb75ec7597464facab4bd3c542141f69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:46.369208 18801 raft_consensus.cc:374] T e0f38a8973a44097a31dea1869c48b74 P e4da7e62b8984ffc8e0f7faa579ef303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:46.369249 18799 raft_consensus.cc:729] T e0f38a8973a44097a31dea1869c48b74 P fb75ec7597464facab4bd3c542141f69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb75ec7597464facab4bd3c542141f69, State: Initialized, Role: FOLLOWER
I20260925 19:08:46.369261 18801 raft_consensus.cc:729] T e0f38a8973a44097a31dea1869c48b74 P e4da7e62b8984ffc8e0f7faa579ef303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4da7e62b8984ffc8e0f7faa579ef303, State: Initialized, Role: FOLLOWER
I20260925 19:08:46.369385 18801 consensus_queue.cc:260] T e0f38a8973a44097a31dea1869c48b74 P e4da7e62b8984ffc8e0f7faa579ef303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4da7e62b8984ffc8e0f7faa579ef303" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 35583 } } peers { permanent_uuid: "cd5b7e5c75c5474aa6f8a23956cbe01d" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 45875 } } peers { permanent_uuid: "fb75ec7597464facab4bd3c542141f69" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 41997 } }
I20260925 19:08:46.369405 18799 consensus_queue.cc:260] T e0f38a8973a44097a31dea1869c48b74 P fb75ec7597464facab4bd3c542141f69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4da7e62b8984ffc8e0f7faa579ef303" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 35583 } } peers { permanent_uuid: "cd5b7e5c75c5474aa6f8a23956cbe01d" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 45875 } } peers { permanent_uuid: "fb75ec7597464facab4bd3c542141f69" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 41997 } }
I20260925 19:08:46.369680 18801 ts_tablet_manager.cc:1428] T e0f38a8973a44097a31dea1869c48b74 P e4da7e62b8984ffc8e0f7faa579ef303: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:08:46.369680 18799 ts_tablet_manager.cc:1428] T e0f38a8973a44097a31dea1869c48b74 P fb75ec7597464facab4bd3c542141f69: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260925 19:08:46.369706 18653 heartbeater.cc:503] Master 127.0.196.254:44059 was elected leader, sending a full tablet report...
I20260925 19:08:46.369834 18522 heartbeater.cc:503] Master 127.0.196.254:44059 was elected leader, sending a full tablet report...
W20260925 19:08:46.465415 18654 tablet.cc:2369] T e0f38a8973a44097a31dea1869c48b74 P e4da7e62b8984ffc8e0f7faa579ef303: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:08:46.501968 18804 raft_consensus.cc:482] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:08:46.502110 18804 raft_consensus.cc:504] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4da7e62b8984ffc8e0f7faa579ef303" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 35583 } } peers { permanent_uuid: "cd5b7e5c75c5474aa6f8a23956cbe01d" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 45875 } } peers { permanent_uuid: "fb75ec7597464facab4bd3c542141f69" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 41997 } }
I20260925 19:08:46.502514 18804 leader_election.cc:288] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4da7e62b8984ffc8e0f7faa579ef303 (127.0.196.194:35583), fb75ec7597464facab4bd3c542141f69 (127.0.196.193:41997)
I20260925 19:08:46.505795 18797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:46.502847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:36685 (local address 127.0.196.193:41997)
0925 19:08:46.502908 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:46.502910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:46.502925 (+ 15us) server_negotiation.cc:407] Connection header received
0925 19:08:46.502950 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:46.502951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:46.502960 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:46.503030 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:46.503221 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.503857 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.505041 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.505414 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.505470 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:46.505477 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:46.505653 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 19:08:46.505701 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 19:08:46.505821 18786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:46.502766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:41997 (local address 127.0.196.195:36685)
0925 19:08:46.502809 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:46.502823 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:46.502873 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:46.503057 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:46.503058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:46.503060 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:46.503142 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:46.503145 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.503936 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.503937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:46.504953 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:46.504955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.505492 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.505493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:46.505573 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:46.505576 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:08:46.505640 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 19:08:46.505713 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":5}
I20260925 19:08:46.505918 18795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:46.502862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:59931 (local address 127.0.196.194:35583)
0925 19:08:46.502914 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:46.502916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:46.503248 (+ 332us) server_negotiation.cc:407] Connection header received
0925 19:08:46.503294 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:46.503295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:46.503303 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:46.503350 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:46.503502 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.504071 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.505067 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.505492 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.505569 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:46.505577 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:46.505787 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 19:08:46.505842 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 19:08:46.506158 18807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:46.502767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:35583 (local address 127.0.196.195:59931)
0925 19:08:46.503153 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:46.503163 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:46.503235 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:46.503370 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:46.503371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:46.503372 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:46.503435 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:46.503437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.504161 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.504161 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:46.504953 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:46.504956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.505591 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.505592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:46.505674 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:46.505678 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:08:46.506020 (+ 342us) client_negotiation.cc:240] Negotiation successful
0925 19:08:46.506072 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":164,"thread_start_us":146,"threads_started":1}
I20260925 19:08:46.506323 18476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f38a8973a44097a31dea1869c48b74" candidate_uuid: "cd5b7e5c75c5474aa6f8a23956cbe01d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb75ec7597464facab4bd3c542141f69" is_pre_election: true
I20260925 19:08:46.506531 18476 raft_consensus.cc:2454] T e0f38a8973a44097a31dea1869c48b74 P fb75ec7597464facab4bd3c542141f69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd5b7e5c75c5474aa6f8a23956cbe01d in term 0.
I20260925 19:08:46.506490 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f38a8973a44097a31dea1869c48b74" candidate_uuid: "cd5b7e5c75c5474aa6f8a23956cbe01d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4da7e62b8984ffc8e0f7faa579ef303" is_pre_election: true
I20260925 19:08:46.506664 18607 raft_consensus.cc:2454] T e0f38a8973a44097a31dea1869c48b74 P e4da7e62b8984ffc8e0f7faa579ef303 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd5b7e5c75c5474aa6f8a23956cbe01d in term 0.
I20260925 19:08:46.506815 18671 leader_election.cc:302] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd5b7e5c75c5474aa6f8a23956cbe01d, fb75ec7597464facab4bd3c542141f69; no voters:
I20260925 19:08:46.506974 18804 raft_consensus.cc:2781] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:08:46.507055 18804 raft_consensus.cc:482] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:08:46.507091 18804 raft_consensus.cc:3037] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:46.507838 18804 raft_consensus.cc:504] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4da7e62b8984ffc8e0f7faa579ef303" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 35583 } } peers { permanent_uuid: "cd5b7e5c75c5474aa6f8a23956cbe01d" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 45875 } } peers { permanent_uuid: "fb75ec7597464facab4bd3c542141f69" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 41997 } }
I20260925 19:08:46.507998 18804 leader_election.cc:288] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d [CANDIDATE]: Term 1 election: Requested vote from peers e4da7e62b8984ffc8e0f7faa579ef303 (127.0.196.194:35583), fb75ec7597464facab4bd3c542141f69 (127.0.196.193:41997)
I20260925 19:08:46.508157 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f38a8973a44097a31dea1869c48b74" candidate_uuid: "cd5b7e5c75c5474aa6f8a23956cbe01d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4da7e62b8984ffc8e0f7faa579ef303"
I20260925 19:08:46.508188 18476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f38a8973a44097a31dea1869c48b74" candidate_uuid: "cd5b7e5c75c5474aa6f8a23956cbe01d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb75ec7597464facab4bd3c542141f69"
I20260925 19:08:46.508263 18476 raft_consensus.cc:3037] T e0f38a8973a44097a31dea1869c48b74 P fb75ec7597464facab4bd3c542141f69 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:46.508255 18607 raft_consensus.cc:3037] T e0f38a8973a44097a31dea1869c48b74 P e4da7e62b8984ffc8e0f7faa579ef303 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:46.509255 18476 raft_consensus.cc:2454] T e0f38a8973a44097a31dea1869c48b74 P fb75ec7597464facab4bd3c542141f69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd5b7e5c75c5474aa6f8a23956cbe01d in term 1.
I20260925 19:08:46.509255 18607 raft_consensus.cc:2454] T e0f38a8973a44097a31dea1869c48b74 P e4da7e62b8984ffc8e0f7faa579ef303 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd5b7e5c75c5474aa6f8a23956cbe01d in term 1.
I20260925 19:08:46.509447 18671 leader_election.cc:302] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd5b7e5c75c5474aa6f8a23956cbe01d, fb75ec7597464facab4bd3c542141f69; no voters:
I20260925 19:08:46.509557 18804 raft_consensus.cc:2781] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:08:46.509703 18804 raft_consensus.cc:686] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d [term 1 LEADER]: Becoming Leader. State: Replica: cd5b7e5c75c5474aa6f8a23956cbe01d, State: Running, Role: LEADER
I20260925 19:08:46.509814 18804 consensus_queue.cc:237] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4da7e62b8984ffc8e0f7faa579ef303" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 35583 } } peers { permanent_uuid: "cd5b7e5c75c5474aa6f8a23956cbe01d" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 45875 } } peers { permanent_uuid: "fb75ec7597464facab4bd3c542141f69" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 41997 } }
I20260925 19:08:46.510744 18333 catalog_manager.cc:5495] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd5b7e5c75c5474aa6f8a23956cbe01d (127.0.196.195). New cstate: current_term: 1 leader_uuid: "cd5b7e5c75c5474aa6f8a23956cbe01d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4da7e62b8984ffc8e0f7faa579ef303" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 35583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd5b7e5c75c5474aa6f8a23956cbe01d" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 45875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb75ec7597464facab4bd3c542141f69" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 41997 } health_report { overall_health: UNKNOWN } } }
I20260925 19:08:46.539896 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:46.537382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48950 (local address 127.0.196.195:45875)
0925 19:08:46.537459 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:46.537461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:46.537470 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:08:46.537522 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:46.537523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:46.537540 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:46.537618 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:08:46.537751 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.538254 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.539089 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:46.539288 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:46.539372 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:46.539543 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:08:46.539643 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:08:46.539735 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:46.539754 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 19:08:46.539772 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 19:08:46.541873 18607 raft_consensus.cc:1261] T e0f38a8973a44097a31dea1869c48b74 P e4da7e62b8984ffc8e0f7faa579ef303 [term 1 FOLLOWER]: Refusing update from remote peer cd5b7e5c75c5474aa6f8a23956cbe01d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:08:46.541910 18476 raft_consensus.cc:1261] T e0f38a8973a44097a31dea1869c48b74 P fb75ec7597464facab4bd3c542141f69 [term 1 FOLLOWER]: Refusing update from remote peer cd5b7e5c75c5474aa6f8a23956cbe01d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:08:46.542236 18808 consensus_queue.cc:1035] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4da7e62b8984ffc8e0f7faa579ef303" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:08:46.542368 18804 consensus_queue.cc:1035] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb75ec7597464facab4bd3c542141f69" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:08:46.545785 18813 mvcc.cc:205] Tried to move back new op lower bound from 7333328185512419328 to 7333328185384943616. Current Snapshot: MvccSnapshot[applied={T|T < 7333328185512419328}]
I20260925 19:08:46.547114 18333 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:08:46.547221 18333 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:08:46.548040 18333 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e5b7766240e4a7087104485e35ead78: Sending DeleteTablet for 3 replicas of tablet e0f38a8973a44097a31dea1869c48b74
I20260925 19:08:46.548406 18718 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f38a8973a44097a31dea1869c48b74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:08:46 UTC) from {username='slave'} at 127.0.0.1:48948
I20260925 19:08:46.548434 18587 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f38a8973a44097a31dea1869c48b74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:08:46 UTC) from {username='slave'} at 127.0.0.1:43620
I20260925 19:08:46.548486 18456 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f38a8973a44097a31dea1869c48b74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:08:46 UTC) from {username='slave'} at 127.0.0.1:44422
I20260925 19:08:46.548687 18821 tablet_replica.cc:330] T e0f38a8973a44097a31dea1869c48b74 P e4da7e62b8984ffc8e0f7faa579ef303: stopping tablet replica
I20260925 19:08:46.548841 18820 tablet_replica.cc:330] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d: stopping tablet replica
I20260925 19:08:46.548863 18821 raft_consensus.cc:2229] T e0f38a8973a44097a31dea1869c48b74 P e4da7e62b8984ffc8e0f7faa579ef303 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:08:46.548935 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 18394
I20260925 19:08:46.548974 18820 raft_consensus.cc:2229] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:08:46.549067 18820 raft_consensus.cc:2258] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:08:46.550195 18820 ts_tablet_manager.cc:1905] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:08:46.549019 18821 raft_consensus.cc:2258] T e0f38a8973a44097a31dea1869c48b74 P e4da7e62b8984ffc8e0f7faa579ef303 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:08:46.552387 18820 ts_tablet_manager.cc:1918] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:08:46.552475 18820 log.cc:1196] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-2/wal/wals/e0f38a8973a44097a31dea1869c48b74
I20260925 19:08:46.552722 18820 ts_tablet_manager.cc:1939] T e0f38a8973a44097a31dea1869c48b74 P cd5b7e5c75c5474aa6f8a23956cbe01d: Deleting consensus metadata
I20260925 19:08:46.553773 18318 catalog_manager.cc:4841] TS cd5b7e5c75c5474aa6f8a23956cbe01d (127.0.196.195:45875): tablet e0f38a8973a44097a31dea1869c48b74 (table test-table [id=5213ee105e3d4e4b90e688f02aeead9a]) successfully deleted
I20260925 19:08:46.554020 18821 ts_tablet_manager.cc:1905] T e0f38a8973a44097a31dea1869c48b74 P e4da7e62b8984ffc8e0f7faa579ef303: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:08:46.555495 18821 ts_tablet_manager.cc:1918] T e0f38a8973a44097a31dea1869c48b74 P e4da7e62b8984ffc8e0f7faa579ef303: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:08:46.555588 18821 log.cc:1196] T e0f38a8973a44097a31dea1869c48b74 P e4da7e62b8984ffc8e0f7faa579ef303: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790363087086945-787-0/minicluster-data/ts-1/wal/wals/e0f38a8973a44097a31dea1869c48b74
I20260925 19:08:46.555917 18821 ts_tablet_manager.cc:1939] T e0f38a8973a44097a31dea1869c48b74 P e4da7e62b8984ffc8e0f7faa579ef303: Deleting consensus metadata
I20260925 19:08:46.556452 18321 catalog_manager.cc:4841] TS e4da7e62b8984ffc8e0f7faa579ef303 (127.0.196.194:35583): tablet e0f38a8973a44097a31dea1869c48b74 (table test-table [id=5213ee105e3d4e4b90e688f02aeead9a]) successfully deleted
W20260925 19:08:46.557205 18318 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.196.193:41997 (error 108)
I20260925 19:08:46.557485 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 18525
I20260925 19:08:46.559309 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:46.559145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:41997 (local address 127.0.0.1:44428)
0925 19:08:46.559211 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:46.559267 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.193:41997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 19:08:46.559465 18318 catalog_manager.cc:4584] TS fb75ec7597464facab4bd3c542141f69 (127.0.196.193:41997): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e0f38a8973a44097a31dea1869c48b74: Network error: Client connection negotiation failed: client connection to 127.0.196.193:41997: connect: Connection refused (error 111)
I20260925 19:08:46.562912 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 18656
I20260925 19:08:46.568583 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 18303
2026-09-25T19:08:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (857 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:08:46 dist-test-slave-rn98 krb5kdc[18826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:08:46 dist-test-slave-rn98 krb5kdc[18826](info): set up 2 sockets
Sep 25 19:08:46 dist-test-slave-rn98 krb5kdc[18826](info): commencing operation
krb5kdc: starting...
W20260925 19:08:47.658454 787 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:08:47 dist-test-slave-rn98 krb5kdc[18826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:08:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:08:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:08:47.781992 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:33949
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:36119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:33949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:08:47.865482 18839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:47.865764 18839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:47.865808 18839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:47.867156 18839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:08:47.867219 18839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:47.867244 18839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:47.867264 18839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:08:47.867296 18839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:08:47.868892 18839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:36119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:33949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:33949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:47.869210 18839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:47.869446 18839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:47.874825 18845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:47.874818 18844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:47.874812 18847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:47.874837 18839 server_base.cc:1034] running on GCE node
I20260925 19:08:47.875401 18839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:47.875737 18839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:47.876914 18839 hybrid_clock.cc:648] HybridClock initialized: now 1790363327876897 us; error 44 us; skew 500 ppm
Sep 25 19:08:47 dist-test-slave-rn98 krb5kdc[18826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:08:47.879688 18839 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:08:47.880259 18839 webserver.cc:460] Webserver started at http://127.0.196.254:40755/ using document root <none> and password file <none>
I20260925 19:08:47.880503 18839 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:47.880573 18839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:47.880738 18839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:47.881939 18839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "382e0b4c28114006a319c6907c79a3c0"
format_stamp: "Formatted at 2026-09-25 19:08:47 on dist-test-slave-rn98"
server_key: "e51265893bbb66aa60889cd3c677705c"
server_key_iv: "ce800c4e4609939c337c67a5f51633b0"
server_key_version: "encryptionkey@0"
I20260925 19:08:47.882393 18839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "382e0b4c28114006a319c6907c79a3c0"
format_stamp: "Formatted at 2026-09-25 19:08:47 on dist-test-slave-rn98"
server_key: "e51265893bbb66aa60889cd3c677705c"
server_key_iv: "ce800c4e4609939c337c67a5f51633b0"
server_key_version: "encryptionkey@0"
I20260925 19:08:47.883996 18839 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260925 19:08:47.884797 18854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:47.884969 18839 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:08:47.885074 18839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "382e0b4c28114006a319c6907c79a3c0"
format_stamp: "Formatted at 2026-09-25 19:08:47 on dist-test-slave-rn98"
server_key: "e51265893bbb66aa60889cd3c677705c"
server_key_iv: "ce800c4e4609939c337c67a5f51633b0"
server_key_version: "encryptionkey@0"
I20260925 19:08:47.885160 18839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:47.926069 18839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:47.928278 18839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:47.928452 18839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:47.934509 18839 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:33949
I20260925 19:08:47.934499 18906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:33949 every 8 connection(s)
I20260925 19:08:47.935158 18839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:08:47.936292 18907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:47.937820 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 18839
I20260925 19:08:47.937945 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:08:47.938170 787 external_mini_cluster.cc:1428] Setting key cf384fa311914c804aa2b6f9ec5d5a76
I20260925 19:08:47.939792 18907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 382e0b4c28114006a319c6907c79a3c0: Bootstrap starting.
I20260925 19:08:47.940604 18907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 382e0b4c28114006a319c6907c79a3c0: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:47.940979 18907 log.cc:824] T 00000000000000000000000000000000 P 382e0b4c28114006a319c6907c79a3c0: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:47.941761 18907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 382e0b4c28114006a319c6907c79a3c0: No bootstrap required, opened a new log
I20260925 19:08:47.944497 18907 raft_consensus.cc:348] T 00000000000000000000000000000000 P 382e0b4c28114006a319c6907c79a3c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382e0b4c28114006a319c6907c79a3c0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33949 } }
I20260925 19:08:47.944695 18907 raft_consensus.cc:374] T 00000000000000000000000000000000 P 382e0b4c28114006a319c6907c79a3c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:47.944741 18907 raft_consensus.cc:729] T 00000000000000000000000000000000 P 382e0b4c28114006a319c6907c79a3c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382e0b4c28114006a319c6907c79a3c0, State: Initialized, Role: FOLLOWER
I20260925 19:08:47.944891 18907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 382e0b4c28114006a319c6907c79a3c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382e0b4c28114006a319c6907c79a3c0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33949 } }
I20260925 19:08:47.944983 18907 raft_consensus.cc:388] T 00000000000000000000000000000000 P 382e0b4c28114006a319c6907c79a3c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:08:47.945024 18907 raft_consensus.cc:482] T 00000000000000000000000000000000 P 382e0b4c28114006a319c6907c79a3c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:08:47.945079 18907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 382e0b4c28114006a319c6907c79a3c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:47.945964 18907 raft_consensus.cc:504] T 00000000000000000000000000000000 P 382e0b4c28114006a319c6907c79a3c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382e0b4c28114006a319c6907c79a3c0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33949 } }
I20260925 19:08:47.946094 18907 leader_election.cc:302] T 00000000000000000000000000000000 P 382e0b4c28114006a319c6907c79a3c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 382e0b4c28114006a319c6907c79a3c0; no voters:
I20260925 19:08:47.946303 18907 leader_election.cc:288] T 00000000000000000000000000000000 P 382e0b4c28114006a319c6907c79a3c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:08:47.946365 18912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 382e0b4c28114006a319c6907c79a3c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:08:47.946594 18907 sys_catalog.cc:564] T 00000000000000000000000000000000 P 382e0b4c28114006a319c6907c79a3c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:08:47.946599 18912 raft_consensus.cc:686] T 00000000000000000000000000000000 P 382e0b4c28114006a319c6907c79a3c0 [term 1 LEADER]: Becoming Leader. State: Replica: 382e0b4c28114006a319c6907c79a3c0, State: Running, Role: LEADER
I20260925 19:08:47.946753 18912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 382e0b4c28114006a319c6907c79a3c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382e0b4c28114006a319c6907c79a3c0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33949 } }
I20260925 19:08:47.947131 18914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 382e0b4c28114006a319c6907c79a3c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 382e0b4c28114006a319c6907c79a3c0. Latest consensus state: current_term: 1 leader_uuid: "382e0b4c28114006a319c6907c79a3c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382e0b4c28114006a319c6907c79a3c0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33949 } } }
I20260925 19:08:47.947135 18913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 382e0b4c28114006a319c6907c79a3c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "382e0b4c28114006a319c6907c79a3c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382e0b4c28114006a319c6907c79a3c0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 33949 } } }
I20260925 19:08:47.947260 18914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 382e0b4c28114006a319c6907c79a3c0 [sys.catalog]: This master's current role is: LEADER
I20260925 19:08:47.947268 18913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 382e0b4c28114006a319c6907c79a3c0 [sys.catalog]: This master's current role is: LEADER
I20260925 19:08:47.948091 18921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:08:47.948784 18921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 19:08:47 dist-test-slave-rn98 krb5kdc[18826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:08:47.951277 18921 catalog_manager.cc:1347] Generated new cluster ID: fc5175f064f6494ab03a0bbc7d876eec
I20260925 19:08:47.951416 18921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:08:47.955582 18910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:47.938991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47412 (local address 127.0.196.254:33949)
0925 19:08:47.939168 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:47.939181 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:47.939359 (+ 178us) server_negotiation.cc:407] Connection header received
0925 19:08:47.939927 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:47.939937 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:47.940011 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:47.940166 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:47.941148 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:47.942587 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:47.943990 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:47.944677 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:47.944793 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:47.950868 (+ 6075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:47.950873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:47.950874 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:08:47.950888 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:47.952998 (+ 2110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:47.953493 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:47.953497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:47.953498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:47.953623 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:47.953917 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:47.953918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:47.953918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:47.954275 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:47.954453 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:47.955156 (+ 703us) server_negotiation.cc:299] Negotiation successful
0925 19:08:47.955334 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260925 19:08:47.968755 18921 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:08:47.969650 18921 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:08:47.973383 18921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 382e0b4c28114006a319c6907c79a3c0: Generated new TSK 0
I20260925 19:08:47.973683 18921 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
I20260925 19:08:48.015143 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:0
--local_ip_for_outbound_sockets=127.0.196.193
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:33949
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:36119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:08:48.119094 18933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:48.119319 18933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:48.119345 18933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:48.120538 18933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:48.120585 18933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:48.120613 18933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:08:48.122185 18933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:36119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:33949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:48.122462 18933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:48.122766 18933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:48.127815 18941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:48.127907 18938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:48.127817 18939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:48.128235 18933 server_base.cc:1034] running on GCE node
I20260925 19:08:48.128463 18933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:48.128681 18933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:48.129829 18933 hybrid_clock.cc:648] HybridClock initialized: now 1790363328129818 us; error 44 us; skew 500 ppm
Sep 25 19:08:48 dist-test-slave-rn98 krb5kdc[18826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:08:48.132439 18933 init.cc:377] Logged in from keytab as kudu/127.0.196.193@KRBTEST.COM (short username kudu)
I20260925 19:08:48.133024 18933 webserver.cc:460] Webserver started at http://127.0.196.193:38361/ using document root <none> and password file <none>
I20260925 19:08:48.133281 18933 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:48.133350 18933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:48.133491 18933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:48.134758 18933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-0/data/instance:
uuid: "c7db5b2efa5e48a09ea4ab7cdb079bdb"
format_stamp: "Formatted at 2026-09-25 19:08:48 on dist-test-slave-rn98"
server_key: "fb3de55b5088ad7c43a3fe67631d7732"
server_key_iv: "0ee8fa25f24e763ce189110baf80b5d7"
server_key_version: "encryptionkey@0"
I20260925 19:08:48.135123 18933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-0/wal/instance:
uuid: "c7db5b2efa5e48a09ea4ab7cdb079bdb"
format_stamp: "Formatted at 2026-09-25 19:08:48 on dist-test-slave-rn98"
server_key: "fb3de55b5088ad7c43a3fe67631d7732"
server_key_iv: "0ee8fa25f24e763ce189110baf80b5d7"
server_key_version: "encryptionkey@0"
I20260925 19:08:48.136487 18933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:08:48.137233 18948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:48.137404 18933 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:08:48.137506 18933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "c7db5b2efa5e48a09ea4ab7cdb079bdb"
format_stamp: "Formatted at 2026-09-25 19:08:48 on dist-test-slave-rn98"
server_key: "fb3de55b5088ad7c43a3fe67631d7732"
server_key_iv: "0ee8fa25f24e763ce189110baf80b5d7"
server_key_version: "encryptionkey@0"
I20260925 19:08:48.137579 18933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:48.175035 18933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:48.177273 18933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:48.177453 18933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:48.177738 18933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:08:48.178068 18933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:08:48.178128 18933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:48.178162 18933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:08:48.178205 18933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:48.184548 18933 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:34795
I20260925 19:08:48.184588 19061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:34795 every 8 connection(s)
I20260925 19:08:48.184948 18933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
I20260925 19:08:48.191491 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 18933
I20260925 19:08:48.191602 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-0/wal/instance
I20260925 19:08:48.191802 787 external_mini_cluster.cc:1428] Setting key d117cf717aa287566989d44d49375d18
Sep 25 19:08:48 dist-test-slave-rn98 krb5kdc[18826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:08:48.196110 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:48.185740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33949 (local address 127.0.196.193:56305)
0925 19:08:48.185891 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:48.185929 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:48.186940 (+ 1011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:48.187192 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:48.187196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:48.187474 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:08:48.187677 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:48.187681 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.188475 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.188476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:48.189557 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:48.189560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.189948 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.189949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:48.190059 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:48.190695 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:08:48.190704 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:08:48.192490 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:08:48.194405 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:48.194415 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:48.194418 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:48.194869 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:48.195071 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:48.195073 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:48.195073 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:48.195217 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:48.195543 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:08:48.195548 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:08:48.195677 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 19:08:48.195796 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 19:08:48.195935 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260925 19:08:48.196102 18910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:48.185753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:56305 (local address 127.0.196.254:33949)
0925 19:08:48.185803 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:48.185805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:48.186013 (+ 208us) server_negotiation.cc:407] Connection header received
0925 19:08:48.187062 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:48.187063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:48.187074 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:48.187134 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:48.187752 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.188376 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.189637 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.189859 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.189918 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:48.192603 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:48.192605 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:48.192605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:08:48.192613 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:48.194312 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:48.194953 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:48.194954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:48.194954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:48.195010 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:48.195318 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:48.195319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:48.195319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:48.195478 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:48.195546 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:48.195839 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 19:08:48.195956 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:08:48.196858 19062 heartbeater.cc:347] Connected to a master server at 127.0.196.254:33949
I20260925 19:08:48.196975 19062 heartbeater.cc:464] Registering TS with master...
I20260925 19:08:48.197176 19062 heartbeater.cc:511] Master 127.0.196.254:33949 requested a full tablet report, sending...
I20260925 19:08:48.197703 18871 ts_manager.cc:194] Registered new tserver with Master: c7db5b2efa5e48a09ea4ab7cdb079bdb (127.0.196.193:34795)
I20260925 19:08:48.198333 18871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.193@KRBTEST.COM'} at 127.0.196.193:56305
WARNING: no policy specified for kudu/127.0.196.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
I20260925 19:08:48.217780 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:0
--local_ip_for_outbound_sockets=127.0.196.194
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:33949
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:36119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:08:48.301888 19067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:48.302191 19067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:48.302240 19067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:48.303536 19067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:48.303586 19067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:48.303614 19067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:08:48.305192 19067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:36119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:33949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:48.305474 19067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:48.305721 19067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:48.311098 19072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:48.311133 19075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:48.311100 19073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:48.311136 19067 server_base.cc:1034] running on GCE node
I20260925 19:08:48.311800 19067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:48.312122 19067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:48.313313 19067 hybrid_clock.cc:648] HybridClock initialized: now 1790363328313314 us; error 62 us; skew 500 ppm
Sep 25 19:08:48 dist-test-slave-rn98 krb5kdc[18826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:08:48.316291 19067 init.cc:377] Logged in from keytab as kudu/127.0.196.194@KRBTEST.COM (short username kudu)
I20260925 19:08:48.316833 19067 webserver.cc:460] Webserver started at http://127.0.196.194:43089/ using document root <none> and password file <none>
I20260925 19:08:48.317081 19067 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:48.317149 19067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:48.317301 19067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:48.318419 19067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-1/data/instance:
uuid: "8979418bfffb46fba70ba21e1782dbc7"
format_stamp: "Formatted at 2026-09-25 19:08:48 on dist-test-slave-rn98"
server_key: "9658940b715f0d06b2679b44a6d20776"
server_key_iv: "db66e51c538e695aba35d6ba86ea2dc3"
server_key_version: "encryptionkey@0"
I20260925 19:08:48.318823 19067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-1/wal/instance:
uuid: "8979418bfffb46fba70ba21e1782dbc7"
format_stamp: "Formatted at 2026-09-25 19:08:48 on dist-test-slave-rn98"
server_key: "9658940b715f0d06b2679b44a6d20776"
server_key_iv: "db66e51c538e695aba35d6ba86ea2dc3"
server_key_version: "encryptionkey@0"
I20260925 19:08:48.320258 19067 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:08:48.321125 19082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:48.321317 19067 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:08:48.321429 19067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "8979418bfffb46fba70ba21e1782dbc7"
format_stamp: "Formatted at 2026-09-25 19:08:48 on dist-test-slave-rn98"
server_key: "9658940b715f0d06b2679b44a6d20776"
server_key_iv: "db66e51c538e695aba35d6ba86ea2dc3"
server_key_version: "encryptionkey@0"
I20260925 19:08:48.321507 19067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:48.332968 19067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:48.335371 19067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:48.335538 19067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:48.335803 19067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:08:48.336128 19067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:08:48.336180 19067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:48.336221 19067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:08:48.336246 19067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:48.342474 19067 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:42489
I20260925 19:08:48.342525 19195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:42489 every 8 connection(s)
I20260925 19:08:48.342952 19067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:08:48 dist-test-slave-rn98 krb5kdc[18826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:08:48.353015 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 19067
I20260925 19:08:48.353142 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-1/wal/instance
I20260925 19:08:48.353375 787 external_mini_cluster.cc:1428] Setting key bc72be215b75272c984db16e8cf82d5c
I20260925 19:08:48.353394 18910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:48.343802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:41991 (local address 127.0.196.254:33949)
0925 19:08:48.343882 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:48.343885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:48.344077 (+ 192us) server_negotiation.cc:407] Connection header received
0925 19:08:48.345020 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:48.345021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:48.345034 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:48.345113 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:48.345775 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.346408 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.347844 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.348043 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.348114 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:48.350519 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:48.350520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:48.350520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:08:48.350528 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:48.351993 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:48.352401 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:48.352402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:48.352402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:48.352457 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:48.352671 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:48.352671 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:48.352672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:48.352803 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:48.352857 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:48.353243 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 19:08:48.353326 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 19:08:48.353668 19198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:48.343748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33949 (local address 127.0.196.194:41991)
0925 19:08:48.343913 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:48.343958 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:48.344898 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:48.345176 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:48.345180 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:48.345459 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:08:48.345668 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:48.345673 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.346532 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.346534 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:48.347741 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:48.347744 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.348128 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.348129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:48.348238 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:48.348810 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:08:48.348820 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:08:48.350423 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:08:48.352073 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:48.352080 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:48.352082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:48.352335 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:48.352512 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:48.352513 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:48.352513 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:48.352622 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:48.352879 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:08:48.352886 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:08:48.353073 (+ 187us) client_negotiation.cc:769] Sending connection context
0925 19:08:48.353204 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 19:08:48.353396 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260925 19:08:48.354223 19196 heartbeater.cc:347] Connected to a master server at 127.0.196.254:33949
I20260925 19:08:48.354346 19196 heartbeater.cc:464] Registering TS with master...
I20260925 19:08:48.354545 19196 heartbeater.cc:511] Master 127.0.196.254:33949 requested a full tablet report, sending...
I20260925 19:08:48.355002 18870 ts_manager.cc:194] Registered new tserver with Master: 8979418bfffb46fba70ba21e1782dbc7 (127.0.196.194:42489)
I20260925 19:08:48.355396 18870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.194@KRBTEST.COM'} at 127.0.196.194:41991
WARNING: no policy specified for kudu/127.0.196.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
I20260925 19:08:48.382275 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:0
--local_ip_for_outbound_sockets=127.0.196.195
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:33949
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:36119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:08:48.467592 19201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:48.467855 19201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:48.467885 19201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:48.469259 19201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:48.469308 19201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:48.469338 19201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:08:48.471395 19201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:36119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:33949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:48.471649 19201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:48.471865 19201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:48.477700 19206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:48.477736 19209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:48.477700 19207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:48.478070 19201 server_base.cc:1034] running on GCE node
I20260925 19:08:48.478278 19201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:48.478515 19201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:48.479696 19201 hybrid_clock.cc:648] HybridClock initialized: now 1790363328479686 us; error 40 us; skew 500 ppm
Sep 25 19:08:48 dist-test-slave-rn98 krb5kdc[18826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:08:48.482831 19201 init.cc:377] Logged in from keytab as kudu/127.0.196.195@KRBTEST.COM (short username kudu)
I20260925 19:08:48.483388 19201 webserver.cc:460] Webserver started at http://127.0.196.195:34225/ using document root <none> and password file <none>
I20260925 19:08:48.483637 19201 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:48.483719 19201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:48.483872 19201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:48.485035 19201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-2/data/instance:
uuid: "ffe838b2011743b7b773564d5e231979"
format_stamp: "Formatted at 2026-09-25 19:08:48 on dist-test-slave-rn98"
server_key: "54bfff54d4c76ee996c860e92697b6b5"
server_key_iv: "f3a1c6b34011abe7b7a4c12630338e6a"
server_key_version: "encryptionkey@0"
I20260925 19:08:48.485452 19201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-2/wal/instance:
uuid: "ffe838b2011743b7b773564d5e231979"
format_stamp: "Formatted at 2026-09-25 19:08:48 on dist-test-slave-rn98"
server_key: "54bfff54d4c76ee996c860e92697b6b5"
server_key_iv: "f3a1c6b34011abe7b7a4c12630338e6a"
server_key_version: "encryptionkey@0"
I20260925 19:08:48.486979 19201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:08:48.487779 19216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:48.487990 19201 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:08:48.488080 19201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "ffe838b2011743b7b773564d5e231979"
format_stamp: "Formatted at 2026-09-25 19:08:48 on dist-test-slave-rn98"
server_key: "54bfff54d4c76ee996c860e92697b6b5"
server_key_iv: "f3a1c6b34011abe7b7a4c12630338e6a"
server_key_version: "encryptionkey@0"
I20260925 19:08:48.488169 19201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:48.502365 19201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:48.504840 19201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:48.504972 19201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:48.505192 19201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:08:48.505646 19201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:08:48.505856 19201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:48.505955 19201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:08:48.505990 19201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:48.512697 19201 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:42529
I20260925 19:08:48.512763 19329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:42529 every 8 connection(s)
I20260925 19:08:48.513135 19201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
I20260925 19:08:48.518083 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 19201
I20260925 19:08:48.518183 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-2/wal/instance
I20260925 19:08:48.518395 787 external_mini_cluster.cc:1428] Setting key 7e95d57efeed44c3bce24ac30cbd9c9f
Sep 25 19:08:48 dist-test-slave-rn98 krb5kdc[18826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:08:48.524503 18910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:48.514070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:60759 (local address 127.0.196.254:33949)
0925 19:08:48.514162 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:48.514164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:48.514447 (+ 283us) server_negotiation.cc:407] Connection header received
0925 19:08:48.515502 (+ 1055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:48.515503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:48.515515 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:48.515584 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:48.516294 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.516831 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.517841 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.518183 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.518268 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:48.520984 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:48.520986 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:48.520986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:08:48.520995 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:48.522508 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:48.523123 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:48.523125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:48.523125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:48.523199 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:48.523621 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:48.523623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:48.523623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:48.523821 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:48.523906 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:48.524258 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 19:08:48.524350 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 19:08:48.524852 19332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:48.514058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:33949 (local address 127.0.196.195:60759)
0925 19:08:48.514284 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:48.514325 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:48.515365 (+ 1040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:48.515682 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:48.515686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:48.515977 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:08:48.516184 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:48.516188 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.516923 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.516924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:48.517741 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:48.517743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.518352 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.518354 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:48.518526 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:48.519304 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:08:48.519316 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:08:48.520860 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:08:48.522680 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:48.522693 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:48.522697 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:48.523006 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:48.523329 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:48.523330 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:48.523330 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:48.523481 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:48.523926 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:08:48.523933 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:08:48.524112 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 19:08:48.524216 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 19:08:48.524570 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":122,"threads_started":1}
I20260925 19:08:48.525432 19330 heartbeater.cc:347] Connected to a master server at 127.0.196.254:33949
I20260925 19:08:48.525532 19330 heartbeater.cc:464] Registering TS with master...
I20260925 19:08:48.525743 19330 heartbeater.cc:511] Master 127.0.196.254:33949 requested a full tablet report, sending...
I20260925 19:08:48.526175 18870 ts_manager.cc:194] Registered new tserver with Master: ffe838b2011743b7b773564d5e231979 (127.0.196.195:42529)
I20260925 19:08:48.526559 18870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.195@KRBTEST.COM'} at 127.0.196.195:60759
I20260925 19:08:48.530362 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:08:48.539229 18910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:48.531847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47414 (local address 127.0.196.254:33949)
0925 19:08:48.531921 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:48.531924 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:48.532002 (+ 78us) server_negotiation.cc:407] Connection header received
0925 19:08:48.532096 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:48.532097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:48.532108 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:48.532155 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:48.532740 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.533241 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.534029 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.534240 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.534342 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:48.536208 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:48.536210 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:48.536210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:08:48.536219 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:48.537716 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:48.538210 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:48.538211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:48.538212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:48.538276 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:48.538515 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:48.538516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:48.538516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:48.538767 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:48.538891 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:48.539049 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 19:08:48.539123 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 19:08:48.541306 18870 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:08:48.543044 18870 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:08:48.550546 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:48.545938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60000 (local address 127.0.196.194:42489)
0925 19:08:48.546780 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:48.546785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:48.546795 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:08:48.546839 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:48.546841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:48.546860 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:48.546940 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:48.547180 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.548284 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.549471 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.550104 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.550178 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:48.550235 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:48.550414 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 19:08:48.550460 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":805,"thread_start_us":120,"threads_started":1}
I20260925 19:08:48.550562 19342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:48.545810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:42489 (local address 127.0.0.1:60000)
0925 19:08:48.546159 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:48.546172 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:48.546235 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:48.547036 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:48.547037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:48.547039 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:48.547081 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:48.547083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.548382 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.548383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:48.549379 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:48.549381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.550196 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.550197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:48.550288 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:48.550304 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 19:08:48.550392 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 19:08:48.550452 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":2,"thread_start_us":115,"threads_started":1}
I20260925 19:08:48.551040 19341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:48.545789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:42529 (local address 127.0.0.1:37268)
0925 19:08:48.546104 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:48.546144 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:48.546211 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:48.546656 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:48.546659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:48.546662 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:48.546787 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:48.546790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.548277 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.548278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:48.549738 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:48.549742 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.550756 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.550757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:48.550846 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:48.550852 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 19:08:48.550890 (+ 38us) client_negotiation.cc:240] Negotiation successful
0925 19:08:48.550939 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":69,"thread_start_us":65,"threads_started":1}
I20260925 19:08:48.551220 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:48.546157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41894 (local address 127.0.196.193:34795)
0925 19:08:48.546375 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:48.546381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:48.546394 (+ 13us) server_negotiation.cc:407] Connection header received
0925 19:08:48.546438 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:48.546440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:48.546455 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:48.546571 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:48.546905 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.548021 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.549848 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.550586 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.550695 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:48.550752 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:48.551090 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 19:08:48.551140 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":110,"threads_started":1}
I20260925 19:08:48.551051 19345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:48.546008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:34795 (local address 127.0.0.1:41894)
0925 19:08:48.546328 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:48.546337 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:48.546374 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:48.546701 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:48.546702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:48.546704 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:48.546787 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:48.546790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.548122 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.548123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:48.549765 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:48.549767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.550785 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.550786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:48.550848 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:48.550853 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:08:48.550900 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 19:08:48.550946 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":135,"thread_start_us":64,"threads_started":1}
I20260925 19:08:48.551463 19343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:48.545905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37268 (local address 127.0.196.195:42529)
0925 19:08:48.546414 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:48.546418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:48.546428 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:08:48.546493 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:48.546496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:48.546513 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:48.546596 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:48.546921 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.548152 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.549849 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.550537 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.551237 (+ 700us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:48.551304 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:48.551331 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 19:08:48.551375 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":95,"threads_started":1}
I20260925 19:08:48.551682 19130 tablet_service.cc:1467] Processing CreateTablet for tablet 485bcefbf609463893a19041e061875d (DEFAULT_TABLE table=test-table [id=7c9708fc11354c65bb000a664eac03d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:08:48.552093 19130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485bcefbf609463893a19041e061875d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:48.552040 19264 tablet_service.cc:1467] Processing CreateTablet for tablet 485bcefbf609463893a19041e061875d (DEFAULT_TABLE table=test-table [id=7c9708fc11354c65bb000a664eac03d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:08:48.552098 18996 tablet_service.cc:1467] Processing CreateTablet for tablet 485bcefbf609463893a19041e061875d (DEFAULT_TABLE table=test-table [id=7c9708fc11354c65bb000a664eac03d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:08:48.552294 19264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485bcefbf609463893a19041e061875d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:48.552417 18996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485bcefbf609463893a19041e061875d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:48.555912 19349 tablet_bootstrap.cc:492] T 485bcefbf609463893a19041e061875d P c7db5b2efa5e48a09ea4ab7cdb079bdb: Bootstrap starting.
I20260925 19:08:48.555912 19348 tablet_bootstrap.cc:492] T 485bcefbf609463893a19041e061875d P 8979418bfffb46fba70ba21e1782dbc7: Bootstrap starting.
I20260925 19:08:48.556190 19347 tablet_bootstrap.cc:492] T 485bcefbf609463893a19041e061875d P ffe838b2011743b7b773564d5e231979: Bootstrap starting.
I20260925 19:08:48.556649 19348 tablet_bootstrap.cc:654] T 485bcefbf609463893a19041e061875d P 8979418bfffb46fba70ba21e1782dbc7: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:48.556649 19349 tablet_bootstrap.cc:654] T 485bcefbf609463893a19041e061875d P c7db5b2efa5e48a09ea4ab7cdb079bdb: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:48.556877 19347 tablet_bootstrap.cc:654] T 485bcefbf609463893a19041e061875d P ffe838b2011743b7b773564d5e231979: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:48.557305 19349 log.cc:824] T 485bcefbf609463893a19041e061875d P c7db5b2efa5e48a09ea4ab7cdb079bdb: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:48.557305 19348 log.cc:824] T 485bcefbf609463893a19041e061875d P 8979418bfffb46fba70ba21e1782dbc7: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:48.557528 19347 log.cc:824] T 485bcefbf609463893a19041e061875d P ffe838b2011743b7b773564d5e231979: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:48.558264 19349 tablet_bootstrap.cc:492] T 485bcefbf609463893a19041e061875d P c7db5b2efa5e48a09ea4ab7cdb079bdb: No bootstrap required, opened a new log
I20260925 19:08:48.558321 19349 ts_tablet_manager.cc:1397] T 485bcefbf609463893a19041e061875d P c7db5b2efa5e48a09ea4ab7cdb079bdb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:08:48.558370 19347 tablet_bootstrap.cc:492] T 485bcefbf609463893a19041e061875d P ffe838b2011743b7b773564d5e231979: No bootstrap required, opened a new log
I20260925 19:08:48.558452 19347 ts_tablet_manager.cc:1397] T 485bcefbf609463893a19041e061875d P ffe838b2011743b7b773564d5e231979: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:08:48.558766 19348 tablet_bootstrap.cc:492] T 485bcefbf609463893a19041e061875d P 8979418bfffb46fba70ba21e1782dbc7: No bootstrap required, opened a new log
I20260925 19:08:48.558892 19348 ts_tablet_manager.cc:1397] T 485bcefbf609463893a19041e061875d P 8979418bfffb46fba70ba21e1782dbc7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 19:08:48.559775 19349 raft_consensus.cc:348] T 485bcefbf609463893a19041e061875d P c7db5b2efa5e48a09ea4ab7cdb079bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979418bfffb46fba70ba21e1782dbc7" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42489 } } peers { permanent_uuid: "ffe838b2011743b7b773564d5e231979" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 42529 } } peers { permanent_uuid: "c7db5b2efa5e48a09ea4ab7cdb079bdb" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34795 } }
I20260925 19:08:48.559926 19349 raft_consensus.cc:374] T 485bcefbf609463893a19041e061875d P c7db5b2efa5e48a09ea4ab7cdb079bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:48.559948 19349 raft_consensus.cc:729] T 485bcefbf609463893a19041e061875d P c7db5b2efa5e48a09ea4ab7cdb079bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7db5b2efa5e48a09ea4ab7cdb079bdb, State: Initialized, Role: FOLLOWER
I20260925 19:08:48.560074 19349 consensus_queue.cc:260] T 485bcefbf609463893a19041e061875d P c7db5b2efa5e48a09ea4ab7cdb079bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979418bfffb46fba70ba21e1782dbc7" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42489 } } peers { permanent_uuid: "ffe838b2011743b7b773564d5e231979" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 42529 } } peers { permanent_uuid: "c7db5b2efa5e48a09ea4ab7cdb079bdb" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34795 } }
I20260925 19:08:48.560395 19062 heartbeater.cc:503] Master 127.0.196.254:33949 was elected leader, sending a full tablet report...
I20260925 19:08:48.560402 19349 ts_tablet_manager.cc:1428] T 485bcefbf609463893a19041e061875d P c7db5b2efa5e48a09ea4ab7cdb079bdb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:08:48.560758 19347 raft_consensus.cc:348] T 485bcefbf609463893a19041e061875d P ffe838b2011743b7b773564d5e231979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979418bfffb46fba70ba21e1782dbc7" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42489 } } peers { permanent_uuid: "ffe838b2011743b7b773564d5e231979" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 42529 } } peers { permanent_uuid: "c7db5b2efa5e48a09ea4ab7cdb079bdb" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34795 } }
I20260925 19:08:48.560956 19347 raft_consensus.cc:374] T 485bcefbf609463893a19041e061875d P ffe838b2011743b7b773564d5e231979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:48.561004 19347 raft_consensus.cc:729] T 485bcefbf609463893a19041e061875d P ffe838b2011743b7b773564d5e231979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffe838b2011743b7b773564d5e231979, State: Initialized, Role: FOLLOWER
I20260925 19:08:48.561146 19347 consensus_queue.cc:260] T 485bcefbf609463893a19041e061875d P ffe838b2011743b7b773564d5e231979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979418bfffb46fba70ba21e1782dbc7" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42489 } } peers { permanent_uuid: "ffe838b2011743b7b773564d5e231979" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 42529 } } peers { permanent_uuid: "c7db5b2efa5e48a09ea4ab7cdb079bdb" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34795 } }
I20260925 19:08:48.561444 19330 heartbeater.cc:503] Master 127.0.196.254:33949 was elected leader, sending a full tablet report...
I20260925 19:08:48.561753 19348 raft_consensus.cc:348] T 485bcefbf609463893a19041e061875d P 8979418bfffb46fba70ba21e1782dbc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979418bfffb46fba70ba21e1782dbc7" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42489 } } peers { permanent_uuid: "ffe838b2011743b7b773564d5e231979" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 42529 } } peers { permanent_uuid: "c7db5b2efa5e48a09ea4ab7cdb079bdb" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34795 } }
I20260925 19:08:48.561942 19348 raft_consensus.cc:374] T 485bcefbf609463893a19041e061875d P 8979418bfffb46fba70ba21e1782dbc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:48.561992 19348 raft_consensus.cc:729] T 485bcefbf609463893a19041e061875d P 8979418bfffb46fba70ba21e1782dbc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8979418bfffb46fba70ba21e1782dbc7, State: Initialized, Role: FOLLOWER
I20260925 19:08:48.561465 19347 ts_tablet_manager.cc:1428] T 485bcefbf609463893a19041e061875d P ffe838b2011743b7b773564d5e231979: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:08:48.562134 19348 consensus_queue.cc:260] T 485bcefbf609463893a19041e061875d P 8979418bfffb46fba70ba21e1782dbc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979418bfffb46fba70ba21e1782dbc7" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42489 } } peers { permanent_uuid: "ffe838b2011743b7b773564d5e231979" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 42529 } } peers { permanent_uuid: "c7db5b2efa5e48a09ea4ab7cdb079bdb" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34795 } }
I20260925 19:08:48.562464 19348 ts_tablet_manager.cc:1428] T 485bcefbf609463893a19041e061875d P 8979418bfffb46fba70ba21e1782dbc7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:08:48.562474 19196 heartbeater.cc:503] Master 127.0.196.254:33949 was elected leader, sending a full tablet report...
W20260925 19:08:48.593542 19197 tablet.cc:2369] T 485bcefbf609463893a19041e061875d P 8979418bfffb46fba70ba21e1782dbc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:08:48.685827 19063 tablet.cc:2369] T 485bcefbf609463893a19041e061875d P c7db5b2efa5e48a09ea4ab7cdb079bdb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:08:48.763857 19331 tablet.cc:2369] T 485bcefbf609463893a19041e061875d P ffe838b2011743b7b773564d5e231979: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:08:48.950300 19353 raft_consensus.cc:482] T 485bcefbf609463893a19041e061875d P c7db5b2efa5e48a09ea4ab7cdb079bdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:08:48.950434 19353 raft_consensus.cc:504] T 485bcefbf609463893a19041e061875d P c7db5b2efa5e48a09ea4ab7cdb079bdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979418bfffb46fba70ba21e1782dbc7" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42489 } } peers { permanent_uuid: "ffe838b2011743b7b773564d5e231979" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 42529 } } peers { permanent_uuid: "c7db5b2efa5e48a09ea4ab7cdb079bdb" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34795 } }
I20260925 19:08:48.950886 19353 leader_election.cc:288] T 485bcefbf609463893a19041e061875d P c7db5b2efa5e48a09ea4ab7cdb079bdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8979418bfffb46fba70ba21e1782dbc7 (127.0.196.194:42489), ffe838b2011743b7b773564d5e231979 (127.0.196.195:42529)
I20260925 19:08:48.953881 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:48.951276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:47913 (local address 127.0.196.194:42489)
0925 19:08:48.951319 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:48.951321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:48.951330 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:08:48.951361 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:48.951362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:48.951372 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:48.951450 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:48.951635 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.952106 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.953105 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.953508 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.953597 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:48.953607 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:48.953770 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 19:08:48.953813 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:08:48.953903 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:48.951084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:42489 (local address 127.0.196.193:47913)
0925 19:08:48.951229 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:48.951241 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:48.951302 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:48.951467 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:48.951468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:48.951470 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:48.951559 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:48.951560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.952194 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.952195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:48.953007 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:48.953009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.953597 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.953598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:48.953673 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:48.953676 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:08:48.953756 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 19:08:48.953800 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":23,"thread_start_us":86,"threads_started":1}
I20260925 19:08:48.953912 19343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:48.951176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:39329 (local address 127.0.196.195:42529)
0925 19:08:48.951249 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:48.951251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:48.951330 (+ 79us) server_negotiation.cc:407] Connection header received
0925 19:08:48.951367 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:48.951368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:48.951378 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:48.951439 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:48.951609 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.952098 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.953095 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.953550 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.953635 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:48.953644 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:48.953799 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 19:08:48.953836 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 19:08:48.953928 19356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:48.951084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:42529 (local address 127.0.196.193:39329)
0925 19:08:48.951225 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:48.951238 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:48.951315 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:48.951445 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:48.951446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:48.951448 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:48.951529 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:48.951531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.952186 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.952188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:48.952997 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:48.952999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.953633 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.953635 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:48.953714 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:48.953717 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:08:48.953781 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 19:08:48.953844 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20260925 19:08:48.954357 19150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485bcefbf609463893a19041e061875d" candidate_uuid: "c7db5b2efa5e48a09ea4ab7cdb079bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8979418bfffb46fba70ba21e1782dbc7" is_pre_election: true
I20260925 19:08:48.954523 19354 raft_consensus.cc:482] T 485bcefbf609463893a19041e061875d P ffe838b2011743b7b773564d5e231979 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:08:48.954561 19150 raft_consensus.cc:2454] T 485bcefbf609463893a19041e061875d P 8979418bfffb46fba70ba21e1782dbc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7db5b2efa5e48a09ea4ab7cdb079bdb in term 0.
I20260925 19:08:48.954494 19284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485bcefbf609463893a19041e061875d" candidate_uuid: "c7db5b2efa5e48a09ea4ab7cdb079bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe838b2011743b7b773564d5e231979" is_pre_election: true
I20260925 19:08:48.954641 19354 raft_consensus.cc:504] T 485bcefbf609463893a19041e061875d P ffe838b2011743b7b773564d5e231979 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979418bfffb46fba70ba21e1782dbc7" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42489 } } peers { permanent_uuid: "ffe838b2011743b7b773564d5e231979" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 42529 } } peers { permanent_uuid: "c7db5b2efa5e48a09ea4ab7cdb079bdb" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34795 } }
I20260925 19:08:48.954802 19284 raft_consensus.cc:2454] T 485bcefbf609463893a19041e061875d P ffe838b2011743b7b773564d5e231979 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7db5b2efa5e48a09ea4ab7cdb079bdb in term 0.
I20260925 19:08:48.954864 18951 leader_election.cc:302] T 485bcefbf609463893a19041e061875d P c7db5b2efa5e48a09ea4ab7cdb079bdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8979418bfffb46fba70ba21e1782dbc7, c7db5b2efa5e48a09ea4ab7cdb079bdb; no voters:
I20260925 19:08:48.954986 19354 leader_election.cc:288] T 485bcefbf609463893a19041e061875d P ffe838b2011743b7b773564d5e231979 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8979418bfffb46fba70ba21e1782dbc7 (127.0.196.194:42489), c7db5b2efa5e48a09ea4ab7cdb079bdb (127.0.196.193:34795)
I20260925 19:08:48.955077 19353 raft_consensus.cc:2781] T 485bcefbf609463893a19041e061875d P c7db5b2efa5e48a09ea4ab7cdb079bdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:08:48.955143 19353 raft_consensus.cc:482] T 485bcefbf609463893a19041e061875d P c7db5b2efa5e48a09ea4ab7cdb079bdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:08:48.955169 19353 raft_consensus.cc:3037] T 485bcefbf609463893a19041e061875d P c7db5b2efa5e48a09ea4ab7cdb079bdb [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:48.956297 19353 raft_consensus.cc:504] T 485bcefbf609463893a19041e061875d P c7db5b2efa5e48a09ea4ab7cdb079bdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979418bfffb46fba70ba21e1782dbc7" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42489 } } peers { permanent_uuid: "ffe838b2011743b7b773564d5e231979" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 42529 } } peers { permanent_uuid: "c7db5b2efa5e48a09ea4ab7cdb079bdb" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34795 } }
I20260925 19:08:48.956637 19150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485bcefbf609463893a19041e061875d" candidate_uuid: "c7db5b2efa5e48a09ea4ab7cdb079bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8979418bfffb46fba70ba21e1782dbc7"
I20260925 19:08:48.956763 19150 raft_consensus.cc:3037] T 485bcefbf609463893a19041e061875d P 8979418bfffb46fba70ba21e1782dbc7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:48.957733 19150 raft_consensus.cc:2454] T 485bcefbf609463893a19041e061875d P 8979418bfffb46fba70ba21e1782dbc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7db5b2efa5e48a09ea4ab7cdb079bdb in term 1.
I20260925 19:08:48.957846 19284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485bcefbf609463893a19041e061875d" candidate_uuid: "c7db5b2efa5e48a09ea4ab7cdb079bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe838b2011743b7b773564d5e231979"
I20260925 19:08:48.957929 19284 raft_consensus.cc:3037] T 485bcefbf609463893a19041e061875d P ffe838b2011743b7b773564d5e231979 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:48.958163 19332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:48.955183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:42489 (local address 127.0.196.195:49965)
0925 19:08:48.955234 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:48.955245 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:48.955283 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:48.955441 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:48.955443 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:48.955445 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:48.955515 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:48.955517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.956400 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.956401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:48.957325 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:48.957328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.957913 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.957914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:48.957998 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:48.958002 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:08:48.958052 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 19:08:48.958091 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 19:08:48.958171 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:48.955251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:49965 (local address 127.0.196.194:42489)
0925 19:08:48.955305 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:48.955308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:48.955316 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:08:48.955337 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:48.955339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:48.955348 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:48.955404 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:48.955581 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.956286 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.957419 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.957775 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.957872 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:48.957882 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:48.958072 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 19:08:48.958117 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 19:08:48.958418 19150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485bcefbf609463893a19041e061875d" candidate_uuid: "ffe838b2011743b7b773564d5e231979" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8979418bfffb46fba70ba21e1782dbc7" is_pre_election: true
I20260925 19:08:48.958508 19150 raft_consensus.cc:2379] T 485bcefbf609463893a19041e061875d P 8979418bfffb46fba70ba21e1782dbc7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ffe838b2011743b7b773564d5e231979 in current term 1: Already voted for candidate c7db5b2efa5e48a09ea4ab7cdb079bdb in this term.
I20260925 19:08:48.958827 19284 raft_consensus.cc:2454] T 485bcefbf609463893a19041e061875d P ffe838b2011743b7b773564d5e231979 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7db5b2efa5e48a09ea4ab7cdb079bdb in term 1.
I20260925 19:08:48.958932 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:48.955493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:44239 (local address 127.0.196.193:34795)
0925 19:08:48.955534 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:48.955536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:48.955684 (+ 148us) server_negotiation.cc:407] Connection header received
0925 19:08:48.955759 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:48.955760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:48.955770 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:48.955831 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:48.955984 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.956635 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.958020 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.958525 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.958611 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:48.958618 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:48.958836 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 19:08:48.958873 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:08:48.958976 18949 leader_election.cc:302] T 485bcefbf609463893a19041e061875d P c7db5b2efa5e48a09ea4ab7cdb079bdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7db5b2efa5e48a09ea4ab7cdb079bdb, ffe838b2011743b7b773564d5e231979; no voters:
I20260925 19:08:48.959049 19353 leader_election.cc:288] T 485bcefbf609463893a19041e061875d P c7db5b2efa5e48a09ea4ab7cdb079bdb [CANDIDATE]: Term 1 election: Requested vote from peers 8979418bfffb46fba70ba21e1782dbc7 (127.0.196.194:42489), ffe838b2011743b7b773564d5e231979 (127.0.196.195:42529)
I20260925 19:08:48.959095 19353 raft_consensus.cc:2781] T 485bcefbf609463893a19041e061875d P c7db5b2efa5e48a09ea4ab7cdb079bdb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:08:48.959122 19358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:48.955429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:34795 (local address 127.0.196.195:44239)
0925 19:08:48.955606 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:48.955615 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:48.955684 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:48.955849 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:48.955850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:48.955852 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:48.955914 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:48.955915 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.956712 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.956713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:48.957890 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:48.957892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.958652 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.958654 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:48.958736 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:48.958742 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 19:08:48.959010 (+ 268us) client_negotiation.cc:240] Negotiation successful
0925 19:08:48.959055 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":105,"threads_started":1}
I20260925 19:08:48.959133 19353 raft_consensus.cc:686] T 485bcefbf609463893a19041e061875d P c7db5b2efa5e48a09ea4ab7cdb079bdb [term 1 LEADER]: Becoming Leader. State: Replica: c7db5b2efa5e48a09ea4ab7cdb079bdb, State: Running, Role: LEADER
I20260925 19:08:48.959340 19353 consensus_queue.cc:237] T 485bcefbf609463893a19041e061875d P c7db5b2efa5e48a09ea4ab7cdb079bdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979418bfffb46fba70ba21e1782dbc7" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42489 } } peers { permanent_uuid: "ffe838b2011743b7b773564d5e231979" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 42529 } } peers { permanent_uuid: "c7db5b2efa5e48a09ea4ab7cdb079bdb" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34795 } }
I20260925 19:08:48.959455 19016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485bcefbf609463893a19041e061875d" candidate_uuid: "ffe838b2011743b7b773564d5e231979" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7db5b2efa5e48a09ea4ab7cdb079bdb" is_pre_election: true
I20260925 19:08:48.959810 19217 leader_election.cc:302] T 485bcefbf609463893a19041e061875d P ffe838b2011743b7b773564d5e231979 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ffe838b2011743b7b773564d5e231979; no voters: 8979418bfffb46fba70ba21e1782dbc7, c7db5b2efa5e48a09ea4ab7cdb079bdb
I20260925 19:08:48.960021 19354 raft_consensus.cc:2726] T 485bcefbf609463893a19041e061875d P ffe838b2011743b7b773564d5e231979 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 19:08:48.960392 18870 catalog_manager.cc:5495] T 485bcefbf609463893a19041e061875d P c7db5b2efa5e48a09ea4ab7cdb079bdb reported cstate change: term changed from 0 to 1, leader changed from <none> to c7db5b2efa5e48a09ea4ab7cdb079bdb (127.0.196.193). New cstate: current_term: 1 leader_uuid: "c7db5b2efa5e48a09ea4ab7cdb079bdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8979418bfffb46fba70ba21e1782dbc7" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffe838b2011743b7b773564d5e231979" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 42529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7db5b2efa5e48a09ea4ab7cdb079bdb" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 34795 } health_report { overall_health: HEALTHY } } }
I20260925 19:08:48.992859 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:48.990173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41898 (local address 127.0.196.193:34795)
0925 19:08:48.990250 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:48.990252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:48.990262 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:08:48.990341 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:48.990342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:48.990369 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:48.990449 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:08:48.990607 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.991219 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.992045 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:48.992251 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:48.992330 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:48.992489 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:08:48.992594 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:08:48.992724 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:48.992754 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 19:08:48.992766 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 19:08:48.995059 19150 raft_consensus.cc:1261] T 485bcefbf609463893a19041e061875d P 8979418bfffb46fba70ba21e1782dbc7 [term 1 FOLLOWER]: Refusing update from remote peer c7db5b2efa5e48a09ea4ab7cdb079bdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:08:48.995079 19284 raft_consensus.cc:1261] T 485bcefbf609463893a19041e061875d P ffe838b2011743b7b773564d5e231979 [term 1 FOLLOWER]: Refusing update from remote peer c7db5b2efa5e48a09ea4ab7cdb079bdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:08:48.995458 19353 consensus_queue.cc:1035] T 485bcefbf609463893a19041e061875d P c7db5b2efa5e48a09ea4ab7cdb079bdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8979418bfffb46fba70ba21e1782dbc7" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 42489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:08:48.995651 19359 consensus_queue.cc:1035] T 485bcefbf609463893a19041e061875d P c7db5b2efa5e48a09ea4ab7cdb079bdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffe838b2011743b7b773564d5e231979" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 42529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:08:48.998299 19362 mvcc.cc:205] Tried to move back new op lower bound from 7333328195560202240 to 7333328195418144768. Current Snapshot: MvccSnapshot[applied={T|T < 7333328195560202240}]
I20260925 19:08:48.999024 19364 mvcc.cc:205] Tried to move back new op lower bound from 7333328195560202240 to 7333328195418144768. Current Snapshot: MvccSnapshot[applied={T|T < 7333328195560202240}]
I20260925 19:08:49.001648 18871 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:08:49.001756 18871 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:08:49.002650 18871 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 382e0b4c28114006a319c6907c79a3c0: Sending DeleteTablet for 3 replicas of tablet 485bcefbf609463893a19041e061875d
I20260925 19:08:49.003072 19264 tablet_service.cc:1514] Processing DeleteTablet for tablet 485bcefbf609463893a19041e061875d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:08:48 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:37268
I20260925 19:08:49.003110 19130 tablet_service.cc:1514] Processing DeleteTablet for tablet 485bcefbf609463893a19041e061875d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:08:48 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:60000
I20260925 19:08:49.003304 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 18933
I20260925 19:08:49.003358 18996 tablet_service.cc:1514] Processing DeleteTablet for tablet 485bcefbf609463893a19041e061875d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:08:48 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:41894
I20260925 19:08:49.004468 19373 tablet_replica.cc:330] T 485bcefbf609463893a19041e061875d P 8979418bfffb46fba70ba21e1782dbc7: stopping tablet replica
I20260925 19:08:49.004644 19373 raft_consensus.cc:2229] T 485bcefbf609463893a19041e061875d P 8979418bfffb46fba70ba21e1782dbc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:08:49.004878 19373 raft_consensus.cc:2258] T 485bcefbf609463893a19041e061875d P 8979418bfffb46fba70ba21e1782dbc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:08:49.005111 19372 tablet_replica.cc:330] T 485bcefbf609463893a19041e061875d P ffe838b2011743b7b773564d5e231979: stopping tablet replica
I20260925 19:08:49.005285 19372 raft_consensus.cc:2229] T 485bcefbf609463893a19041e061875d P ffe838b2011743b7b773564d5e231979 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:08:49.005435 19372 raft_consensus.cc:2258] T 485bcefbf609463893a19041e061875d P ffe838b2011743b7b773564d5e231979 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:08:49.006034 19372 ts_tablet_manager.cc:1905] T 485bcefbf609463893a19041e061875d P ffe838b2011743b7b773564d5e231979: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:08:49.006258 19373 ts_tablet_manager.cc:1905] T 485bcefbf609463893a19041e061875d P 8979418bfffb46fba70ba21e1782dbc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:08:49.008291 19372 ts_tablet_manager.cc:1918] T 485bcefbf609463893a19041e061875d P ffe838b2011743b7b773564d5e231979: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:08:49.008385 19372 log.cc:1196] T 485bcefbf609463893a19041e061875d P ffe838b2011743b7b773564d5e231979: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-2/wal/wals/485bcefbf609463893a19041e061875d
I20260925 19:08:49.008648 19372 ts_tablet_manager.cc:1939] T 485bcefbf609463893a19041e061875d P ffe838b2011743b7b773564d5e231979: Deleting consensus metadata
I20260925 19:08:49.009114 18855 catalog_manager.cc:4841] TS ffe838b2011743b7b773564d5e231979 (127.0.196.195:42529): tablet 485bcefbf609463893a19041e061875d (table test-table [id=7c9708fc11354c65bb000a664eac03d8]) successfully deleted
I20260925 19:08:49.009380 19373 ts_tablet_manager.cc:1918] T 485bcefbf609463893a19041e061875d P 8979418bfffb46fba70ba21e1782dbc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:08:49.009460 19373 log.cc:1196] T 485bcefbf609463893a19041e061875d P 8979418bfffb46fba70ba21e1782dbc7: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790363087086945-787-0/minicluster-data/ts-1/wal/wals/485bcefbf609463893a19041e061875d
I20260925 19:08:49.009704 19373 ts_tablet_manager.cc:1939] T 485bcefbf609463893a19041e061875d P 8979418bfffb46fba70ba21e1782dbc7: Deleting consensus metadata
W20260925 19:08:49.010445 18855 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.196.193:34795 (error 108)
I20260925 19:08:49.010644 18857 catalog_manager.cc:4841] TS 8979418bfffb46fba70ba21e1782dbc7 (127.0.196.194:42489): tablet 485bcefbf609463893a19041e061875d (table test-table [id=7c9708fc11354c65bb000a664eac03d8]) successfully deleted
I20260925 19:08:49.010816 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 19067
I20260925 19:08:49.011464 19342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:49.011018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:34795 (local address 127.0.0.1:41902)
0925 19:08:49.011399 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:49.011436 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.193:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362}
W20260925 19:08:49.012437 18855 catalog_manager.cc:4584] TS c7db5b2efa5e48a09ea4ab7cdb079bdb (127.0.196.193:34795): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 485bcefbf609463893a19041e061875d: Network error: Client connection negotiation failed: client connection to 127.0.196.193:34795: connect: Connection refused (error 111)
I20260925 19:08:49.016065 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 19201
I20260925 19:08:49.021318 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 18839
2026-09-25T19:08:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2453 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T19:08:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:08:49Z Disabled control of system clock
I20260925 19:08:49.049844 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:41977
--webserver_interface=127.0.196.254
--webserver_port=0
--builtin_ntp_servers=127.0.196.212:41973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:41977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 19:08:49.138547 19378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:49.138859 19378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:49.138883 19378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:49.140168 19378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:08:49.140224 19378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:49.140237 19378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:49.140250 19378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:08:49.140306 19378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:08:49.141726 19378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:41973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:41977
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:41977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:49.142009 19378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:49.142331 19378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:49.147514 19383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:49.147562 19378 server_base.cc:1034] running on GCE node
W20260925 19:08:49.147523 19386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:49.147523 19384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:49.148020 19378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:49.148351 19378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:49.149536 19378 hybrid_clock.cc:648] HybridClock initialized: now 1790363329149530 us; error 39 us; skew 500 ppm
I20260925 19:08:49.154938 19378 webserver.cc:460] Webserver started at http://127.0.196.254:38839/ using document root <none> and password file <none>
I20260925 19:08:49.155217 19378 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:49.155292 19378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:49.155457 19378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:49.156623 19378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "c08a12e6f2214294ad3c9b55c7b4a2bd"
format_stamp: "Formatted at 2026-09-25 19:08:49 on dist-test-slave-rn98"
server_key: "4a67b8a0f22f23ab5d7335a6755c239c"
server_key_iv: "4a2e2644d29c853381669e992fb0f705"
server_key_version: "encryptionkey@0"
I20260925 19:08:49.157019 19378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "c08a12e6f2214294ad3c9b55c7b4a2bd"
format_stamp: "Formatted at 2026-09-25 19:08:49 on dist-test-slave-rn98"
server_key: "4a67b8a0f22f23ab5d7335a6755c239c"
server_key_iv: "4a2e2644d29c853381669e992fb0f705"
server_key_version: "encryptionkey@0"
I20260925 19:08:49.158406 19378 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:08:49.159276 19392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:49.159451 19378 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:08:49.159562 19378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "c08a12e6f2214294ad3c9b55c7b4a2bd"
format_stamp: "Formatted at 2026-09-25 19:08:49 on dist-test-slave-rn98"
server_key: "4a67b8a0f22f23ab5d7335a6755c239c"
server_key_iv: "4a2e2644d29c853381669e992fb0f705"
server_key_version: "encryptionkey@0"
I20260925 19:08:49.159652 19378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:49.169355 19378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:49.169791 19378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:49.169972 19378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:49.174163 19378 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:41977
I20260925 19:08:49.174214 19444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:41977 every 8 connection(s)
I20260925 19:08:49.174669 19378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:08:49.175071 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 19378
I20260925 19:08:49.175217 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:08:49.175412 787 external_mini_cluster.cc:1428] Setting key 604d928ad805098177591f8c5f7609b6
I20260925 19:08:49.175938 19445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:49.179494 19445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c08a12e6f2214294ad3c9b55c7b4a2bd: Bootstrap starting.
I20260925 19:08:49.180229 19445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c08a12e6f2214294ad3c9b55c7b4a2bd: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:49.180573 19445 log.cc:824] T 00000000000000000000000000000000 P c08a12e6f2214294ad3c9b55c7b4a2bd: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:49.181385 19445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c08a12e6f2214294ad3c9b55c7b4a2bd: No bootstrap required, opened a new log
I20260925 19:08:49.181792 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:49.176004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49944 (local address 127.0.196.254:41977)
0925 19:08:49.176225 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:49.176235 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:49.176269 (+ 34us) server_negotiation.cc:407] Connection header received
0925 19:08:49.176856 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:49.176866 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:49.176975 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:49.177140 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:49.177364 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.178775 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.179569 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.180204 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.180342 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:49.181144 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:49.181153 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:49.181154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:08:49.181327 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:49.181361 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:08:49.181372 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:49.181447 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:49.181524 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 19:08:49.181609 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
I20260925 19:08:49.183645 19445 raft_consensus.cc:348] T 00000000000000000000000000000000 P c08a12e6f2214294ad3c9b55c7b4a2bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08a12e6f2214294ad3c9b55c7b4a2bd" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41977 } }
I20260925 19:08:49.183833 19445 raft_consensus.cc:374] T 00000000000000000000000000000000 P c08a12e6f2214294ad3c9b55c7b4a2bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:49.183883 19445 raft_consensus.cc:729] T 00000000000000000000000000000000 P c08a12e6f2214294ad3c9b55c7b4a2bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c08a12e6f2214294ad3c9b55c7b4a2bd, State: Initialized, Role: FOLLOWER
I20260925 19:08:49.184017 19445 consensus_queue.cc:260] T 00000000000000000000000000000000 P c08a12e6f2214294ad3c9b55c7b4a2bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08a12e6f2214294ad3c9b55c7b4a2bd" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41977 } }
I20260925 19:08:49.184099 19445 raft_consensus.cc:388] T 00000000000000000000000000000000 P c08a12e6f2214294ad3c9b55c7b4a2bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:08:49.184142 19445 raft_consensus.cc:482] T 00000000000000000000000000000000 P c08a12e6f2214294ad3c9b55c7b4a2bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:08:49.184192 19445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c08a12e6f2214294ad3c9b55c7b4a2bd [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:49.184851 19445 raft_consensus.cc:504] T 00000000000000000000000000000000 P c08a12e6f2214294ad3c9b55c7b4a2bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08a12e6f2214294ad3c9b55c7b4a2bd" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41977 } }
I20260925 19:08:49.184984 19445 leader_election.cc:302] T 00000000000000000000000000000000 P c08a12e6f2214294ad3c9b55c7b4a2bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c08a12e6f2214294ad3c9b55c7b4a2bd; no voters:
I20260925 19:08:49.185217 19445 leader_election.cc:288] T 00000000000000000000000000000000 P c08a12e6f2214294ad3c9b55c7b4a2bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:08:49.185250 19450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c08a12e6f2214294ad3c9b55c7b4a2bd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:08:49.185369 19450 raft_consensus.cc:686] T 00000000000000000000000000000000 P c08a12e6f2214294ad3c9b55c7b4a2bd [term 1 LEADER]: Becoming Leader. State: Replica: c08a12e6f2214294ad3c9b55c7b4a2bd, State: Running, Role: LEADER
I20260925 19:08:49.185463 19450 consensus_queue.cc:237] T 00000000000000000000000000000000 P c08a12e6f2214294ad3c9b55c7b4a2bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08a12e6f2214294ad3c9b55c7b4a2bd" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41977 } }
I20260925 19:08:49.185534 19445 sys_catalog.cc:564] T 00000000000000000000000000000000 P c08a12e6f2214294ad3c9b55c7b4a2bd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:08:49.185883 19452 sys_catalog.cc:455] T 00000000000000000000000000000000 P c08a12e6f2214294ad3c9b55c7b4a2bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c08a12e6f2214294ad3c9b55c7b4a2bd. Latest consensus state: current_term: 1 leader_uuid: "c08a12e6f2214294ad3c9b55c7b4a2bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08a12e6f2214294ad3c9b55c7b4a2bd" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41977 } } }
I20260925 19:08:49.185891 19451 sys_catalog.cc:455] T 00000000000000000000000000000000 P c08a12e6f2214294ad3c9b55c7b4a2bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c08a12e6f2214294ad3c9b55c7b4a2bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08a12e6f2214294ad3c9b55c7b4a2bd" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41977 } } }
I20260925 19:08:49.186077 19452 sys_catalog.cc:458] T 00000000000000000000000000000000 P c08a12e6f2214294ad3c9b55c7b4a2bd [sys.catalog]: This master's current role is: LEADER
I20260925 19:08:49.186096 19451 sys_catalog.cc:458] T 00000000000000000000000000000000 P c08a12e6f2214294ad3c9b55c7b4a2bd [sys.catalog]: This master's current role is: LEADER
I20260925 19:08:49.186479 19459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:08:49.187044 19459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:08:49.188630 19459 catalog_manager.cc:1347] Generated new cluster ID: a79aa9ca2f2e42ba8e39da8fd2fb121a
I20260925 19:08:49.188706 19459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:08:49.198743 19459 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:08:49.199260 19459 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:08:49.206837 19459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c08a12e6f2214294ad3c9b55c7b4a2bd: Generated new TSK 0
I20260925 19:08:49.207082 19459 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:08:49.218004 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:0
--local_ip_for_outbound_sockets=127.0.196.193
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:41977
--builtin_ntp_servers=127.0.196.212:41973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 19:08:49.302521 19469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:49.302868 19469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:49.302914 19469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:49.304203 19469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:49.304280 19469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:49.304322 19469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:08:49.305754 19469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:41973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:41977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:49.306041 19469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:49.306365 19469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:49.312628 19474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:49.312646 19475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:49.312647 19477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:49.313095 19469 server_base.cc:1034] running on GCE node
I20260925 19:08:49.313354 19469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:49.313634 19469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:49.314812 19469 hybrid_clock.cc:648] HybridClock initialized: now 1790363329314790 us; error 44 us; skew 500 ppm
I20260925 19:08:49.316186 19469 webserver.cc:460] Webserver started at http://127.0.196.193:46835/ using document root <none> and password file <none>
I20260925 19:08:49.316486 19469 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:49.316563 19469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:49.316717 19469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:49.317862 19469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-0/data/instance:
uuid: "dee7dbeb965c4d2ab39bea6cac7feb12"
format_stamp: "Formatted at 2026-09-25 19:08:49 on dist-test-slave-rn98"
server_key: "884f354328c5b458967a49dad683feeb"
server_key_iv: "604b86f4544487a7944eacbafcbf12e0"
server_key_version: "encryptionkey@0"
I20260925 19:08:49.318256 19469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-0/wal/instance:
uuid: "dee7dbeb965c4d2ab39bea6cac7feb12"
format_stamp: "Formatted at 2026-09-25 19:08:49 on dist-test-slave-rn98"
server_key: "884f354328c5b458967a49dad683feeb"
server_key_iv: "604b86f4544487a7944eacbafcbf12e0"
server_key_version: "encryptionkey@0"
I20260925 19:08:49.319769 19469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:08:49.320648 19483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:49.320832 19469 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:08:49.320943 19469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "dee7dbeb965c4d2ab39bea6cac7feb12"
format_stamp: "Formatted at 2026-09-25 19:08:49 on dist-test-slave-rn98"
server_key: "884f354328c5b458967a49dad683feeb"
server_key_iv: "604b86f4544487a7944eacbafcbf12e0"
server_key_version: "encryptionkey@0"
I20260925 19:08:49.321022 19469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:49.330862 19469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:49.331213 19469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:49.331385 19469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:49.331635 19469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:08:49.331993 19469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:08:49.332047 19469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:49.332083 19469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:08:49.332129 19469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:49.337895 19469 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:36283
I20260925 19:08:49.337978 19596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:36283 every 8 connection(s)
I20260925 19:08:49.338342 19469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
I20260925 19:08:49.343461 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 19469
I20260925 19:08:49.343587 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-0/wal/instance
I20260925 19:08:49.343796 787 external_mini_cluster.cc:1428] Setting key a2651f6902ef9e72bc5063f0fca9d4c1
I20260925 19:08:49.344007 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:49.339238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:58367 (local address 127.0.196.254:41977)
0925 19:08:49.339310 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:49.339312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:49.339542 (+ 230us) server_negotiation.cc:407] Connection header received
0925 19:08:49.340029 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:49.340030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:49.340043 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:49.340114 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:49.340774 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.341275 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.342266 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.342456 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.342511 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:49.343503 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:49.343505 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:49.343505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:08:49.343512 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:49.343530 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:08:49.343535 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:49.343608 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:49.343827 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 19:08:49.343893 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 19:08:49.344194 19599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:49.339188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41977 (local address 127.0.196.193:58367)
0925 19:08:49.339403 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:49.339446 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:49.339943 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:49.340173 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:49.340176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:49.340184 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:08:49.340698 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:49.340703 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.341359 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.341360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:49.342188 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:49.342191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.342531 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.342532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:49.342704 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:49.343322 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:08:49.343322 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:08:49.343355 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:08:49.343380 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:08:49.343640 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:08:49.343649 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:08:49.343653 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:08:49.343827 (+ 174us) client_negotiation.cc:240] Negotiation successful
0925 19:08:49.343955 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":116,"threads_started":1}
I20260925 19:08:49.344817 19597 heartbeater.cc:347] Connected to a master server at 127.0.196.254:41977
I20260925 19:08:49.344906 19597 heartbeater.cc:464] Registering TS with master...
I20260925 19:08:49.344908 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:0
--local_ip_for_outbound_sockets=127.0.196.194
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:41977
--builtin_ntp_servers=127.0.196.212:41973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 19:08:49.345130 19597 heartbeater.cc:511] Master 127.0.196.254:41977 requested a full tablet report, sending...
I20260925 19:08:49.345717 19409 ts_manager.cc:194] Registered new tserver with Master: dee7dbeb965c4d2ab39bea6cac7feb12 (127.0.196.193:36283)
I20260925 19:08:49.346287 19409 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.196.193:58367
W20260925 19:08:49.429354 19600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:49.429579 19600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:49.429603 19600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:49.430881 19600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:49.430929 19600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:49.430958 19600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:08:49.432397 19600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:41973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:41977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:49.432665 19600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:49.432981 19600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:49.438112 19606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:49.438206 19600 server_base.cc:1034] running on GCE node
W20260925 19:08:49.438089 19605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:49.438279 19608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:49.438510 19600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:49.438843 19600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:49.440021 19600 hybrid_clock.cc:648] HybridClock initialized: now 1790363329439987 us; error 45 us; skew 500 ppm
I20260925 19:08:49.441313 19600 webserver.cc:460] Webserver started at http://127.0.196.194:35295/ using document root <none> and password file <none>
I20260925 19:08:49.441589 19600 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:49.441658 19600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:49.441836 19600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:49.443069 19600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-1/data/instance:
uuid: "b4263dac7d584547902a0386e2ddd429"
format_stamp: "Formatted at 2026-09-25 19:08:49 on dist-test-slave-rn98"
server_key: "0a88a8661ca00087c05b8b839aca7292"
server_key_iv: "f2d82ab035e3dc87678b73573007fd4b"
server_key_version: "encryptionkey@0"
I20260925 19:08:49.443444 19600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-1/wal/instance:
uuid: "b4263dac7d584547902a0386e2ddd429"
format_stamp: "Formatted at 2026-09-25 19:08:49 on dist-test-slave-rn98"
server_key: "0a88a8661ca00087c05b8b839aca7292"
server_key_iv: "f2d82ab035e3dc87678b73573007fd4b"
server_key_version: "encryptionkey@0"
I20260925 19:08:49.445616 19600 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260925 19:08:49.446748 19614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:49.446938 19600 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:08:49.447053 19600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "b4263dac7d584547902a0386e2ddd429"
format_stamp: "Formatted at 2026-09-25 19:08:49 on dist-test-slave-rn98"
server_key: "0a88a8661ca00087c05b8b839aca7292"
server_key_iv: "f2d82ab035e3dc87678b73573007fd4b"
server_key_version: "encryptionkey@0"
I20260925 19:08:49.447140 19600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:49.468083 19600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:49.468433 19600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:49.468587 19600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:49.468847 19600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:08:49.469195 19600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:08:49.469250 19600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:49.469296 19600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:08:49.469321 19600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:49.475374 19600 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:36667
I20260925 19:08:49.475419 19727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:36667 every 8 connection(s)
I20260925 19:08:49.475804 19600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
I20260925 19:08:49.480389 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 19600
I20260925 19:08:49.480510 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-1/wal/instance
I20260925 19:08:49.480763 787 external_mini_cluster.cc:1428] Setting key 20a2824c368a2aadea71a1a9b0e058b8
I20260925 19:08:49.481678 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:49.476804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:49861 (local address 127.0.196.254:41977)
0925 19:08:49.476874 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:49.476876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:49.477071 (+ 195us) server_negotiation.cc:407] Connection header received
0925 19:08:49.477549 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:49.477551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:49.477560 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:49.477623 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:49.478314 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.478823 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.479834 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.480028 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.480109 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:49.481363 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:49.481364 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:49.481364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:08:49.481369 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:49.481387 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:08:49.481392 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:49.481454 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:49.481558 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 19:08:49.481587 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 19:08:49.481967 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:0
--local_ip_for_outbound_sockets=127.0.196.195
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:41977
--builtin_ntp_servers=127.0.196.212:41973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 19:08:49.482199 19730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:49.476727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41977 (local address 127.0.196.194:49861)
0925 19:08:49.476923 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:49.476970 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:49.477465 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:49.477679 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:49.477683 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:49.477691 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:08:49.478202 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:49.478207 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.478922 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.478922 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:49.479739 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:49.479741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.480182 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.480183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:49.480380 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:49.481196 (+ 816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:08:49.481197 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:08:49.481235 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:08:49.481261 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:08:49.481460 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:08:49.481467 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:08:49.481471 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:08:49.481546 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 19:08:49.481668 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":101,"threads_started":1}
I20260925 19:08:49.482730 19728 heartbeater.cc:347] Connected to a master server at 127.0.196.254:41977
I20260925 19:08:49.482838 19728 heartbeater.cc:464] Registering TS with master...
I20260925 19:08:49.483055 19728 heartbeater.cc:511] Master 127.0.196.254:41977 requested a full tablet report, sending...
I20260925 19:08:49.483440 19409 ts_manager.cc:194] Registered new tserver with Master: b4263dac7d584547902a0386e2ddd429 (127.0.196.194:36667)
I20260925 19:08:49.483829 19409 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.196.194:49861
W20260925 19:08:49.566361 19731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:49.566591 19731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:49.566670 19731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:49.567906 19731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:49.567956 19731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:49.567982 19731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:08:49.569326 19731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:41973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:41977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:49.569550 19731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:49.569833 19731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:49.575714 19739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:49.575707 19737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:49.575702 19736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:49.576105 19731 server_base.cc:1034] running on GCE node
I20260925 19:08:49.576311 19731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:49.576598 19731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:49.577800 19731 hybrid_clock.cc:648] HybridClock initialized: now 1790363329577779 us; error 41 us; skew 500 ppm
I20260925 19:08:49.579331 19731 webserver.cc:460] Webserver started at http://127.0.196.195:41015/ using document root <none> and password file <none>
I20260925 19:08:49.579620 19731 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:49.579687 19731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:49.579826 19731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:49.580971 19731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-2/data/instance:
uuid: "d47fe2fbd5d84d0cb4890c08664637b5"
format_stamp: "Formatted at 2026-09-25 19:08:49 on dist-test-slave-rn98"
server_key: "c15346af2618134a813f9e0b5b0c1823"
server_key_iv: "f73e0942461345f65c2b4c4f44600d40"
server_key_version: "encryptionkey@0"
I20260925 19:08:49.581357 19731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-2/wal/instance:
uuid: "d47fe2fbd5d84d0cb4890c08664637b5"
format_stamp: "Formatted at 2026-09-25 19:08:49 on dist-test-slave-rn98"
server_key: "c15346af2618134a813f9e0b5b0c1823"
server_key_iv: "f73e0942461345f65c2b4c4f44600d40"
server_key_version: "encryptionkey@0"
I20260925 19:08:49.582962 19731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:08:49.584056 19745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:49.584260 19731 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:08:49.584383 19731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "d47fe2fbd5d84d0cb4890c08664637b5"
format_stamp: "Formatted at 2026-09-25 19:08:49 on dist-test-slave-rn98"
server_key: "c15346af2618134a813f9e0b5b0c1823"
server_key_iv: "f73e0942461345f65c2b4c4f44600d40"
server_key_version: "encryptionkey@0"
I20260925 19:08:49.584478 19731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:49.606108 19731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:49.606494 19731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:49.606761 19731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:49.607029 19731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:08:49.607445 19731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:08:49.607515 19731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:49.607556 19731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:08:49.607591 19731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:49.613752 19731 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:42517
I20260925 19:08:49.613801 19858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:42517 every 8 connection(s)
I20260925 19:08:49.614149 19731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
I20260925 19:08:49.617671 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 19731
I20260925 19:08:49.617815 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-2/wal/instance
I20260925 19:08:49.618031 787 external_mini_cluster.cc:1428] Setting key eb796c850c323960ab15b42171263209
I20260925 19:08:49.620463 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:49.615017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:47345 (local address 127.0.196.254:41977)
0925 19:08:49.615092 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:49.615096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:49.615408 (+ 312us) server_negotiation.cc:407] Connection header received
0925 19:08:49.615924 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:49.615925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:49.615937 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:49.615996 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:49.616710 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.617205 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.618691 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.619004 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.619064 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:49.620157 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:49.620159 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:49.620160 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:08:49.620164 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:49.620180 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:08:49.620185 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:49.620235 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:49.620373 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 19:08:49.620399 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 19:08:49.620682 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:49.614995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41977 (local address 127.0.196.195:47345)
0925 19:08:49.615255 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:49.615294 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:49.615809 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:49.616065 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:49.616069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:49.616078 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:08:49.616610 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:49.616615 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.617349 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.617350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:49.618580 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:49.618584 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.619108 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.619110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:49.619309 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:49.619981 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:08:49.619982 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:08:49.620016 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:08:49.620041 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:08:49.620263 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:08:49.620268 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:08:49.620271 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:08:49.620337 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 19:08:49.620430 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":101,"threads_started":1}
I20260925 19:08:49.621263 19859 heartbeater.cc:347] Connected to a master server at 127.0.196.254:41977
I20260925 19:08:49.621384 19859 heartbeater.cc:464] Registering TS with master...
I20260925 19:08:49.621605 19859 heartbeater.cc:511] Master 127.0.196.254:41977 requested a full tablet report, sending...
I20260925 19:08:49.622045 19409 ts_manager.cc:194] Registered new tserver with Master: d47fe2fbd5d84d0cb4890c08664637b5 (127.0.196.195:42517)
I20260925 19:08:49.622397 19409 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.196.195:47345
I20260925 19:08:49.630250 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:08:49.634944 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:49.631785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49948 (local address 127.0.196.254:41977)
0925 19:08:49.631898 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:49.631900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:49.631915 (+ 15us) server_negotiation.cc:407] Connection header received
0925 19:08:49.631988 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:49.631989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:49.631998 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:49.632047 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:49.632214 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.632709 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.633460 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.633644 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.633699 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:49.634502 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:49.634503 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:49.634504 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:08:49.634509 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:49.634528 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:08:49.634537 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:49.634662 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:49.634802 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 19:08:49.634830 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 19:08:49.636656 19409 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:08:49.638234 19409 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:08:49.645176 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:49.640996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32846 (local address 127.0.196.195:42517)
0925 19:08:49.641184 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:49.641188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:49.641252 (+ 64us) server_negotiation.cc:407] Connection header received
0925 19:08:49.641377 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:49.641380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:49.641395 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:49.641471 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:49.641772 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.642953 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.644243 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.644754 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.644856 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:49.644955 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:49.645042 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 19:08:49.645081 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20260925 19:08:49.645310 19870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:49.640893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45328 (local address 127.0.196.194:36667)
0925 19:08:49.641069 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:49.641075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:49.641270 (+ 195us) server_negotiation.cc:407] Connection header received
0925 19:08:49.641380 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:49.641382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:49.641399 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:49.641481 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:49.641773 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.642607 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.644060 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.644814 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.644891 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:49.644982 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:49.645185 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 19:08:49.645227 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":101,"threads_started":1}
I20260925 19:08:49.645380 19871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:49.640971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44922 (local address 127.0.196.193:36283)
0925 19:08:49.641174 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:49.641179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:49.641561 (+ 382us) server_negotiation.cc:407] Connection header received
0925 19:08:49.641621 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:49.641623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:49.641644 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:49.641730 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:49.641932 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.643046 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.644165 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.644860 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.644916 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:49.644986 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:49.645232 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 19:08:49.645274 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":81,"threads_started":1}
I20260925 19:08:49.645169 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:49.640644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:42517 (local address 127.0.0.1:32846)
0925 19:08:49.641095 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:49.641181 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:49.641277 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:49.641562 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:49.641566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:49.641569 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:49.641678 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:49.641681 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.643059 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.643060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:49.644154 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:49.644156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.644858 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.644859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:49.644939 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:49.644952 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 19:08:49.645028 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 19:08:49.645084 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":73,"spinlock_wait_cycles":384,"thread_start_us":135,"threads_started":1}
I20260925 19:08:49.645597 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:49.640644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:36667 (local address 127.0.0.1:45328)
0925 19:08:49.641193 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:49.641204 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:49.641296 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:49.641563 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:49.641566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:49.641572 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:49.641660 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:49.641663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.642741 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.642742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:49.643965 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:49.643968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.645023 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.645024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:49.645087 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:49.645093 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 19:08:49.645430 (+ 337us) client_negotiation.cc:240] Negotiation successful
0925 19:08:49.645482 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":4,"thread_start_us":132,"threads_started":1}
I20260925 19:08:49.645522 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:49.640664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:36283 (local address 127.0.0.1:44922)
0925 19:08:49.641406 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:49.641415 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:49.641461 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:49.641813 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:49.641814 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:49.641815 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:49.641846 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:49.641848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.643107 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.643108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:49.644069 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:49.644072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.645021 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.645023 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:49.645087 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:49.645094 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 19:08:49.645143 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 19:08:49.645189 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":4,"thread_start_us":117,"threads_started":1}
I20260925 19:08:49.646303 19662 tablet_service.cc:1467] Processing CreateTablet for tablet 6f2f7cdcdeb848f388f34aed350db837 (DEFAULT_TABLE table=test-table [id=82d1c2ec5a734aa69fca467cecaf446b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:08:49.646379 19793 tablet_service.cc:1467] Processing CreateTablet for tablet 6f2f7cdcdeb848f388f34aed350db837 (DEFAULT_TABLE table=test-table [id=82d1c2ec5a734aa69fca467cecaf446b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:08:49.646764 19793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f2f7cdcdeb848f388f34aed350db837. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:49.646780 19662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f2f7cdcdeb848f388f34aed350db837. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:49.649653 19873 tablet_bootstrap.cc:492] T 6f2f7cdcdeb848f388f34aed350db837 P b4263dac7d584547902a0386e2ddd429: Bootstrap starting.
I20260925 19:08:49.649678 19874 tablet_bootstrap.cc:492] T 6f2f7cdcdeb848f388f34aed350db837 P d47fe2fbd5d84d0cb4890c08664637b5: Bootstrap starting.
I20260925 19:08:49.650292 19873 tablet_bootstrap.cc:654] T 6f2f7cdcdeb848f388f34aed350db837 P b4263dac7d584547902a0386e2ddd429: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:49.650194 19531 tablet_service.cc:1467] Processing CreateTablet for tablet 6f2f7cdcdeb848f388f34aed350db837 (DEFAULT_TABLE table=test-table [id=82d1c2ec5a734aa69fca467cecaf446b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:08:49.650512 19874 tablet_bootstrap.cc:654] T 6f2f7cdcdeb848f388f34aed350db837 P d47fe2fbd5d84d0cb4890c08664637b5: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:49.650548 19531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f2f7cdcdeb848f388f34aed350db837. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:49.650696 19873 log.cc:824] T 6f2f7cdcdeb848f388f34aed350db837 P b4263dac7d584547902a0386e2ddd429: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:49.650899 19874 log.cc:824] T 6f2f7cdcdeb848f388f34aed350db837 P d47fe2fbd5d84d0cb4890c08664637b5: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:49.651614 19874 tablet_bootstrap.cc:492] T 6f2f7cdcdeb848f388f34aed350db837 P d47fe2fbd5d84d0cb4890c08664637b5: No bootstrap required, opened a new log
I20260925 19:08:49.651626 19873 tablet_bootstrap.cc:492] T 6f2f7cdcdeb848f388f34aed350db837 P b4263dac7d584547902a0386e2ddd429: No bootstrap required, opened a new log
I20260925 19:08:49.651703 19873 ts_tablet_manager.cc:1397] T 6f2f7cdcdeb848f388f34aed350db837 P b4263dac7d584547902a0386e2ddd429: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 19:08:49.651705 19874 ts_tablet_manager.cc:1397] T 6f2f7cdcdeb848f388f34aed350db837 P d47fe2fbd5d84d0cb4890c08664637b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:08:49.653790 19874 raft_consensus.cc:348] T 6f2f7cdcdeb848f388f34aed350db837 P d47fe2fbd5d84d0cb4890c08664637b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4263dac7d584547902a0386e2ddd429" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 36667 } } peers { permanent_uuid: "d47fe2fbd5d84d0cb4890c08664637b5" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 42517 } } peers { permanent_uuid: "dee7dbeb965c4d2ab39bea6cac7feb12" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36283 } }
I20260925 19:08:49.653962 19874 raft_consensus.cc:374] T 6f2f7cdcdeb848f388f34aed350db837 P d47fe2fbd5d84d0cb4890c08664637b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:49.654006 19874 raft_consensus.cc:729] T 6f2f7cdcdeb848f388f34aed350db837 P d47fe2fbd5d84d0cb4890c08664637b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d47fe2fbd5d84d0cb4890c08664637b5, State: Initialized, Role: FOLLOWER
I20260925 19:08:49.654167 19874 consensus_queue.cc:260] T 6f2f7cdcdeb848f388f34aed350db837 P d47fe2fbd5d84d0cb4890c08664637b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4263dac7d584547902a0386e2ddd429" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 36667 } } peers { permanent_uuid: "d47fe2fbd5d84d0cb4890c08664637b5" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 42517 } } peers { permanent_uuid: "dee7dbeb965c4d2ab39bea6cac7feb12" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36283 } }
I20260925 19:08:49.654325 19873 raft_consensus.cc:348] T 6f2f7cdcdeb848f388f34aed350db837 P b4263dac7d584547902a0386e2ddd429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4263dac7d584547902a0386e2ddd429" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 36667 } } peers { permanent_uuid: "d47fe2fbd5d84d0cb4890c08664637b5" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 42517 } } peers { permanent_uuid: "dee7dbeb965c4d2ab39bea6cac7feb12" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36283 } }
I20260925 19:08:49.654495 19873 raft_consensus.cc:374] T 6f2f7cdcdeb848f388f34aed350db837 P b4263dac7d584547902a0386e2ddd429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:49.654497 19874 ts_tablet_manager.cc:1428] T 6f2f7cdcdeb848f388f34aed350db837 P d47fe2fbd5d84d0cb4890c08664637b5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:08:49.654541 19859 heartbeater.cc:503] Master 127.0.196.254:41977 was elected leader, sending a full tablet report...
I20260925 19:08:49.654543 19873 raft_consensus.cc:729] T 6f2f7cdcdeb848f388f34aed350db837 P b4263dac7d584547902a0386e2ddd429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4263dac7d584547902a0386e2ddd429, State: Initialized, Role: FOLLOWER
I20260925 19:08:49.654688 19873 consensus_queue.cc:260] T 6f2f7cdcdeb848f388f34aed350db837 P b4263dac7d584547902a0386e2ddd429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4263dac7d584547902a0386e2ddd429" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 36667 } } peers { permanent_uuid: "d47fe2fbd5d84d0cb4890c08664637b5" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 42517 } } peers { permanent_uuid: "dee7dbeb965c4d2ab39bea6cac7feb12" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36283 } }
I20260925 19:08:49.654958 19873 ts_tablet_manager.cc:1428] T 6f2f7cdcdeb848f388f34aed350db837 P b4263dac7d584547902a0386e2ddd429: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 19:08:49.655025 19728 heartbeater.cc:503] Master 127.0.196.254:41977 was elected leader, sending a full tablet report...
I20260925 19:08:49.655182 19877 tablet_bootstrap.cc:492] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12: Bootstrap starting.
I20260925 19:08:49.655956 19877 tablet_bootstrap.cc:654] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:49.656267 19877 log.cc:824] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:49.657270 19877 tablet_bootstrap.cc:492] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12: No bootstrap required, opened a new log
I20260925 19:08:49.657343 19877 ts_tablet_manager.cc:1397] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:08:49.659389 19877 raft_consensus.cc:348] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4263dac7d584547902a0386e2ddd429" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 36667 } } peers { permanent_uuid: "d47fe2fbd5d84d0cb4890c08664637b5" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 42517 } } peers { permanent_uuid: "dee7dbeb965c4d2ab39bea6cac7feb12" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36283 } }
I20260925 19:08:49.659540 19877 raft_consensus.cc:374] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:49.659564 19877 raft_consensus.cc:729] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee7dbeb965c4d2ab39bea6cac7feb12, State: Initialized, Role: FOLLOWER
I20260925 19:08:49.659689 19877 consensus_queue.cc:260] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4263dac7d584547902a0386e2ddd429" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 36667 } } peers { permanent_uuid: "d47fe2fbd5d84d0cb4890c08664637b5" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 42517 } } peers { permanent_uuid: "dee7dbeb965c4d2ab39bea6cac7feb12" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36283 } }
I20260925 19:08:49.659971 19877 ts_tablet_manager.cc:1428] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:08:49.660102 19597 heartbeater.cc:503] Master 127.0.196.254:41977 was elected leader, sending a full tablet report...
W20260925 19:08:49.726415 19729 tablet.cc:2369] T 6f2f7cdcdeb848f388f34aed350db837 P b4263dac7d584547902a0386e2ddd429: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:08:49.839151 19598 tablet.cc:2369] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:08:49.864728 19860 tablet.cc:2369] T 6f2f7cdcdeb848f388f34aed350db837 P d47fe2fbd5d84d0cb4890c08664637b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:08:49.951743 19881 raft_consensus.cc:482] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:08:49.951947 19881 raft_consensus.cc:504] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4263dac7d584547902a0386e2ddd429" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 36667 } } peers { permanent_uuid: "d47fe2fbd5d84d0cb4890c08664637b5" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 42517 } } peers { permanent_uuid: "dee7dbeb965c4d2ab39bea6cac7feb12" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36283 } }
I20260925 19:08:49.952409 19881 leader_election.cc:288] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4263dac7d584547902a0386e2ddd429 (127.0.196.194:36667), d47fe2fbd5d84d0cb4890c08664637b5 (127.0.196.195:42517)
I20260925 19:08:49.955724 19870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:49.952638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:39547 (local address 127.0.196.194:36667)
0925 19:08:49.952703 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:49.952705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:49.952863 (+ 158us) server_negotiation.cc:407] Connection header received
0925 19:08:49.952965 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:49.952966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:49.952979 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:49.953029 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:49.953229 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.953734 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.954957 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.955312 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.955371 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:49.955402 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:49.955597 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 19:08:49.955644 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 19:08:49.955717 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:49.952719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:48411 (local address 127.0.196.195:42517)
0925 19:08:49.952770 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:49.952772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:49.952881 (+ 109us) server_negotiation.cc:407] Connection header received
0925 19:08:49.952972 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:49.952973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:49.952983 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:49.953066 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:49.953281 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.953810 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.954882 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.955319 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.955419 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:49.955428 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:49.955578 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 19:08:49.955623 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:08:49.955729 19882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:49.952619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:42517 (local address 127.0.196.193:48411)
0925 19:08:49.952787 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:49.952798 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:49.952895 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:49.953088 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:49.953089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:49.953092 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:49.953167 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:49.953169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.953905 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.953906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:49.954788 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:49.954791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.955433 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.955434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:49.955499 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:49.955502 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:08:49.955572 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 19:08:49.955619 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":124,"threads_started":1}
I20260925 19:08:49.955740 19883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:49.952596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:36667 (local address 127.0.196.193:39547)
0925 19:08:49.952760 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:49.952773 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:49.952888 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:49.953057 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:49.953058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:49.953061 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:49.953137 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:49.953140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.953827 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.953829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:49.954849 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:49.954852 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.955408 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.955410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:49.955489 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:49.955494 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:08:49.955589 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 19:08:49.955637 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":90,"threads_started":1}
I20260925 19:08:49.956382 19813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f2f7cdcdeb848f388f34aed350db837" candidate_uuid: "dee7dbeb965c4d2ab39bea6cac7feb12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d47fe2fbd5d84d0cb4890c08664637b5" is_pre_election: true
I20260925 19:08:49.956434 19682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f2f7cdcdeb848f388f34aed350db837" candidate_uuid: "dee7dbeb965c4d2ab39bea6cac7feb12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4263dac7d584547902a0386e2ddd429" is_pre_election: true
I20260925 19:08:49.956594 19682 raft_consensus.cc:2454] T 6f2f7cdcdeb848f388f34aed350db837 P b4263dac7d584547902a0386e2ddd429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dee7dbeb965c4d2ab39bea6cac7feb12 in term 0.
I20260925 19:08:49.956583 19813 raft_consensus.cc:2454] T 6f2f7cdcdeb848f388f34aed350db837 P d47fe2fbd5d84d0cb4890c08664637b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dee7dbeb965c4d2ab39bea6cac7feb12 in term 0.
I20260925 19:08:49.956858 19487 leader_election.cc:302] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4263dac7d584547902a0386e2ddd429, dee7dbeb965c4d2ab39bea6cac7feb12; no voters:
I20260925 19:08:49.957011 19881 raft_consensus.cc:2781] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:08:49.957077 19881 raft_consensus.cc:482] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:08:49.957103 19881 raft_consensus.cc:3037] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:49.957906 19881 raft_consensus.cc:504] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4263dac7d584547902a0386e2ddd429" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 36667 } } peers { permanent_uuid: "d47fe2fbd5d84d0cb4890c08664637b5" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 42517 } } peers { permanent_uuid: "dee7dbeb965c4d2ab39bea6cac7feb12" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36283 } }
I20260925 19:08:49.958081 19881 leader_election.cc:288] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12 [CANDIDATE]: Term 1 election: Requested vote from peers b4263dac7d584547902a0386e2ddd429 (127.0.196.194:36667), d47fe2fbd5d84d0cb4890c08664637b5 (127.0.196.195:42517)
I20260925 19:08:49.958290 19682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f2f7cdcdeb848f388f34aed350db837" candidate_uuid: "dee7dbeb965c4d2ab39bea6cac7feb12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4263dac7d584547902a0386e2ddd429"
I20260925 19:08:49.958401 19682 raft_consensus.cc:3037] T 6f2f7cdcdeb848f388f34aed350db837 P b4263dac7d584547902a0386e2ddd429 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:49.959283 19682 raft_consensus.cc:2454] T 6f2f7cdcdeb848f388f34aed350db837 P b4263dac7d584547902a0386e2ddd429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dee7dbeb965c4d2ab39bea6cac7feb12 in term 1.
I20260925 19:08:49.959452 19487 leader_election.cc:302] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4263dac7d584547902a0386e2ddd429, dee7dbeb965c4d2ab39bea6cac7feb12; no voters:
I20260925 19:08:49.959431 19813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f2f7cdcdeb848f388f34aed350db837" candidate_uuid: "dee7dbeb965c4d2ab39bea6cac7feb12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d47fe2fbd5d84d0cb4890c08664637b5"
I20260925 19:08:49.959563 19881 raft_consensus.cc:2781] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:08:49.959542 19813 raft_consensus.cc:3037] T 6f2f7cdcdeb848f388f34aed350db837 P d47fe2fbd5d84d0cb4890c08664637b5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:49.959797 19881 raft_consensus.cc:686] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12 [term 1 LEADER]: Becoming Leader. State: Replica: dee7dbeb965c4d2ab39bea6cac7feb12, State: Running, Role: LEADER
I20260925 19:08:49.959906 19881 consensus_queue.cc:237] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4263dac7d584547902a0386e2ddd429" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 36667 } } peers { permanent_uuid: "d47fe2fbd5d84d0cb4890c08664637b5" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 42517 } } peers { permanent_uuid: "dee7dbeb965c4d2ab39bea6cac7feb12" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36283 } }
I20260925 19:08:49.960516 19813 raft_consensus.cc:2454] T 6f2f7cdcdeb848f388f34aed350db837 P d47fe2fbd5d84d0cb4890c08664637b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dee7dbeb965c4d2ab39bea6cac7feb12 in term 1.
I20260925 19:08:49.960865 19408 catalog_manager.cc:5495] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12 reported cstate change: term changed from 0 to 1, leader changed from <none> to dee7dbeb965c4d2ab39bea6cac7feb12 (127.0.196.193). New cstate: current_term: 1 leader_uuid: "dee7dbeb965c4d2ab39bea6cac7feb12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4263dac7d584547902a0386e2ddd429" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 36667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d47fe2fbd5d84d0cb4890c08664637b5" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 42517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dee7dbeb965c4d2ab39bea6cac7feb12" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 36283 } health_report { overall_health: HEALTHY } } }
I20260925 19:08:50.000097 19871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:49.997477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44938 (local address 127.0.196.193:36283)
0925 19:08:49.997543 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:49.997546 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:49.997556 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:08:49.997649 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:49.997650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:49.997671 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:49.997774 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:08:49.997928 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.998408 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.999287 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:49.999480 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:49.999587 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:49.999761 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:08:49.999850 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:08:49.999967 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:49.999997 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 19:08:50.000009 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 19:08:50.002138 19682 raft_consensus.cc:1261] T 6f2f7cdcdeb848f388f34aed350db837 P b4263dac7d584547902a0386e2ddd429 [term 1 FOLLOWER]: Refusing update from remote peer dee7dbeb965c4d2ab39bea6cac7feb12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:08:50.002138 19813 raft_consensus.cc:1261] T 6f2f7cdcdeb848f388f34aed350db837 P d47fe2fbd5d84d0cb4890c08664637b5 [term 1 FOLLOWER]: Refusing update from remote peer dee7dbeb965c4d2ab39bea6cac7feb12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:08:50.002555 19884 consensus_queue.cc:1035] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4263dac7d584547902a0386e2ddd429" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 36667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:08:50.002732 19881 consensus_queue.cc:1035] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d47fe2fbd5d84d0cb4890c08664637b5" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:08:50.005457 19888 mvcc.cc:205] Tried to move back new op lower bound from 7333328199685668864 to 7333328199516483584. Current Snapshot: MvccSnapshot[applied={T|T < 7333328199685668864}]
I20260925 19:08:50.008191 19408 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:08:50.008374 19408 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:08:50.009902 19408 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c08a12e6f2214294ad3c9b55c7b4a2bd: Sending DeleteTablet for 3 replicas of tablet 6f2f7cdcdeb848f388f34aed350db837
I20260925 19:08:50.010263 19662 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f2f7cdcdeb848f388f34aed350db837 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:08:50 UTC) from {username='slave'} at 127.0.0.1:45328
I20260925 19:08:50.010320 19531 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f2f7cdcdeb848f388f34aed350db837 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:08:50 UTC) from {username='slave'} at 127.0.0.1:44922
I20260925 19:08:50.010584 19897 tablet_replica.cc:330] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12: stopping tablet replica
I20260925 19:08:50.010716 19897 raft_consensus.cc:2229] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:08:50.010826 19897 raft_consensus.cc:2258] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:08:50.011900 19793 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f2f7cdcdeb848f388f34aed350db837 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:08:50 UTC) from {username='slave'} at 127.0.0.1:32846
I20260925 19:08:50.012183 19898 tablet_replica.cc:330] T 6f2f7cdcdeb848f388f34aed350db837 P d47fe2fbd5d84d0cb4890c08664637b5: stopping tablet replica
I20260925 19:08:50.012303 19898 raft_consensus.cc:2229] T 6f2f7cdcdeb848f388f34aed350db837 P d47fe2fbd5d84d0cb4890c08664637b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:08:50.012434 19898 raft_consensus.cc:2258] T 6f2f7cdcdeb848f388f34aed350db837 P d47fe2fbd5d84d0cb4890c08664637b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:08:50.012641 19897 ts_tablet_manager.cc:1905] T 6f2f7cdcdeb848f388f34aed350db837 P dee7dbeb965c4d2ab39bea6cac7feb12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:08:50.012727 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 19469
I20260925 19:08:50.015627 19896 tablet_replica.cc:330] T 6f2f7cdcdeb848f388f34aed350db837 P b4263dac7d584547902a0386e2ddd429: stopping tablet replica
I20260925 19:08:50.015741 19898 ts_tablet_manager.cc:1905] T 6f2f7cdcdeb848f388f34aed350db837 P d47fe2fbd5d84d0cb4890c08664637b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:08:50.015776 19896 raft_consensus.cc:2229] T 6f2f7cdcdeb848f388f34aed350db837 P b4263dac7d584547902a0386e2ddd429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:08:50.015893 19896 raft_consensus.cc:2258] T 6f2f7cdcdeb848f388f34aed350db837 P b4263dac7d584547902a0386e2ddd429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:08:50.016388 19896 ts_tablet_manager.cc:1905] T 6f2f7cdcdeb848f388f34aed350db837 P b4263dac7d584547902a0386e2ddd429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:08:50.017716 19898 ts_tablet_manager.cc:1918] T 6f2f7cdcdeb848f388f34aed350db837 P d47fe2fbd5d84d0cb4890c08664637b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:08:50.017802 19898 log.cc:1196] T 6f2f7cdcdeb848f388f34aed350db837 P d47fe2fbd5d84d0cb4890c08664637b5: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-2/wal/wals/6f2f7cdcdeb848f388f34aed350db837
I20260925 19:08:50.018070 19898 ts_tablet_manager.cc:1939] T 6f2f7cdcdeb848f388f34aed350db837 P d47fe2fbd5d84d0cb4890c08664637b5: Deleting consensus metadata
I20260925 19:08:50.018261 19896 ts_tablet_manager.cc:1918] T 6f2f7cdcdeb848f388f34aed350db837 P b4263dac7d584547902a0386e2ddd429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:08:50.018334 19896 log.cc:1196] T 6f2f7cdcdeb848f388f34aed350db837 P b4263dac7d584547902a0386e2ddd429: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790363087086945-787-0/minicluster-data/ts-1/wal/wals/6f2f7cdcdeb848f388f34aed350db837
I20260925 19:08:50.018594 19896 ts_tablet_manager.cc:1939] T 6f2f7cdcdeb848f388f34aed350db837 P b4263dac7d584547902a0386e2ddd429: Deleting consensus metadata
I20260925 19:08:50.018657 19393 catalog_manager.cc:4841] TS d47fe2fbd5d84d0cb4890c08664637b5 (127.0.196.195:42517): tablet 6f2f7cdcdeb848f388f34aed350db837 (table test-table [id=82d1c2ec5a734aa69fca467cecaf446b]) successfully deleted
I20260925 19:08:50.019166 19396 catalog_manager.cc:4841] TS b4263dac7d584547902a0386e2ddd429 (127.0.196.194:36667): tablet 6f2f7cdcdeb848f388f34aed350db837 (table test-table [id=82d1c2ec5a734aa69fca467cecaf446b]) successfully deleted
W20260925 19:08:50.019222 19394 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.196.193:36283 (error 108)
I20260925 19:08:50.019451 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 19600
I20260925 19:08:50.020114 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:50.019550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:36283 (local address 127.0.0.1:44944)
0925 19:08:50.020044 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:50.020085 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.193:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471}
W20260925 19:08:50.020254 19394 catalog_manager.cc:4584] TS dee7dbeb965c4d2ab39bea6cac7feb12 (127.0.196.193:36283): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6f2f7cdcdeb848f388f34aed350db837: Network error: Client connection negotiation failed: client connection to 127.0.196.193:36283: connect: Connection refused (error 111)
I20260925 19:08:50.025470 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 19731
I20260925 19:08:50.030268 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 19378
2026-09-25T19:08:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1007 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:08:50 dist-test-slave-rn98 krb5kdc[19902](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:08:50 dist-test-slave-rn98 krb5kdc[19902](info): set up 2 sockets
Sep 25 19:08:50 dist-test-slave-rn98 krb5kdc[19902](info): commencing operation
krb5kdc: starting...
W20260925 19:08:51.086777 787 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:08:51 dist-test-slave-rn98 krb5kdc[19902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:08:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:08:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:08:51.214738 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:41731
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:40149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:41731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:08:51.297626 19915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:51.297904 19915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:51.297947 19915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:51.299256 19915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:08:51.299333 19915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:51.299358 19915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:51.299381 19915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:08:51.299410 19915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:08:51.301019 19915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:40149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:41731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:41731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:51.301347 19915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:51.301606 19915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:51.307081 19923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:51.307085 19921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:51.307096 19920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:51.307395 19915 server_base.cc:1034] running on GCE node
I20260925 19:08:51.307672 19915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:51.307971 19915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:51.309122 19915 hybrid_clock.cc:648] HybridClock initialized: now 1790363331309092 us; error 48 us; skew 500 ppm
Sep 25 19:08:51 dist-test-slave-rn98 krb5kdc[19902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:08:51.312408 19915 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:08:51.312988 19915 webserver.cc:460] Webserver started at http://127.0.196.254:33599/ using document root <none> and password file <none>
I20260925 19:08:51.313262 19915 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:51.313313 19915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:51.313470 19915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:51.314723 19915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "6ade36a50b8b47d5a84290fccdf30ec1"
format_stamp: "Formatted at 2026-09-25 19:08:51 on dist-test-slave-rn98"
server_key: "0d1e13fbd11843f4377a8088f105cf5a"
server_key_iv: "bc6060a4e9638c8fa1ddbdeda005c4e1"
server_key_version: "encryptionkey@0"
I20260925 19:08:51.315135 19915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "6ade36a50b8b47d5a84290fccdf30ec1"
format_stamp: "Formatted at 2026-09-25 19:08:51 on dist-test-slave-rn98"
server_key: "0d1e13fbd11843f4377a8088f105cf5a"
server_key_iv: "bc6060a4e9638c8fa1ddbdeda005c4e1"
server_key_version: "encryptionkey@0"
I20260925 19:08:51.316601 19915 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 19:08:51.317454 19930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:51.317646 19915 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:08:51.317746 19915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "6ade36a50b8b47d5a84290fccdf30ec1"
format_stamp: "Formatted at 2026-09-25 19:08:51 on dist-test-slave-rn98"
server_key: "0d1e13fbd11843f4377a8088f105cf5a"
server_key_iv: "bc6060a4e9638c8fa1ddbdeda005c4e1"
server_key_version: "encryptionkey@0"
I20260925 19:08:51.317812 19915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:51.327251 19915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:51.329387 19915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:51.329596 19915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:51.333736 19915 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:41731
I20260925 19:08:51.333828 19982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:41731 every 8 connection(s)
I20260925 19:08:51.334188 19915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:08:51.335230 19983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:51.337620 19983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ade36a50b8b47d5a84290fccdf30ec1: Bootstrap starting.
I20260925 19:08:51.338269 19983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ade36a50b8b47d5a84290fccdf30ec1: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:51.338572 19983 log.cc:824] T 00000000000000000000000000000000 P 6ade36a50b8b47d5a84290fccdf30ec1: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:51.339351 19983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ade36a50b8b47d5a84290fccdf30ec1: No bootstrap required, opened a new log
I20260925 19:08:51.339818 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 19915
I20260925 19:08:51.339939 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:08:51.340122 787 external_mini_cluster.cc:1428] Setting key 273439d1fb3269de1d50aaa2db2fe570
I20260925 19:08:51.341743 19983 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ade36a50b8b47d5a84290fccdf30ec1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ade36a50b8b47d5a84290fccdf30ec1" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41731 } }
I20260925 19:08:51.341935 19983 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ade36a50b8b47d5a84290fccdf30ec1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:51.341976 19983 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ade36a50b8b47d5a84290fccdf30ec1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ade36a50b8b47d5a84290fccdf30ec1, State: Initialized, Role: FOLLOWER
I20260925 19:08:51.342115 19983 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ade36a50b8b47d5a84290fccdf30ec1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ade36a50b8b47d5a84290fccdf30ec1" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41731 } }
I20260925 19:08:51.342202 19983 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ade36a50b8b47d5a84290fccdf30ec1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:08:51.342237 19983 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ade36a50b8b47d5a84290fccdf30ec1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:08:51.342288 19983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ade36a50b8b47d5a84290fccdf30ec1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:51.343528 19983 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ade36a50b8b47d5a84290fccdf30ec1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ade36a50b8b47d5a84290fccdf30ec1" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41731 } }
I20260925 19:08:51.343645 19983 leader_election.cc:302] T 00000000000000000000000000000000 P 6ade36a50b8b47d5a84290fccdf30ec1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ade36a50b8b47d5a84290fccdf30ec1; no voters:
I20260925 19:08:51.343833 19983 leader_election.cc:288] T 00000000000000000000000000000000 P 6ade36a50b8b47d5a84290fccdf30ec1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:08:51.343895 19988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ade36a50b8b47d5a84290fccdf30ec1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:08:51.344044 19988 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ade36a50b8b47d5a84290fccdf30ec1 [term 1 LEADER]: Becoming Leader. State: Replica: 6ade36a50b8b47d5a84290fccdf30ec1, State: Running, Role: LEADER
I20260925 19:08:51.344074 19983 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ade36a50b8b47d5a84290fccdf30ec1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:08:51.344137 19988 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ade36a50b8b47d5a84290fccdf30ec1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ade36a50b8b47d5a84290fccdf30ec1" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41731 } }
I20260925 19:08:51.344477 19989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ade36a50b8b47d5a84290fccdf30ec1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ade36a50b8b47d5a84290fccdf30ec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ade36a50b8b47d5a84290fccdf30ec1" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41731 } } }
I20260925 19:08:51.344575 19989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ade36a50b8b47d5a84290fccdf30ec1 [sys.catalog]: This master's current role is: LEADER
I20260925 19:08:51.344762 19990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ade36a50b8b47d5a84290fccdf30ec1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ade36a50b8b47d5a84290fccdf30ec1. Latest consensus state: current_term: 1 leader_uuid: "6ade36a50b8b47d5a84290fccdf30ec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ade36a50b8b47d5a84290fccdf30ec1" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 41731 } } }
I20260925 19:08:51.344846 19990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ade36a50b8b47d5a84290fccdf30ec1 [sys.catalog]: This master's current role is: LEADER
I20260925 19:08:51.345120 19994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:08:51.345649 19994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:08:51.347543 19994 catalog_manager.cc:1347] Generated new cluster ID: 1905ea050fe64c1393d497abdb080017
I20260925 19:08:51.347600 19994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:08:51 dist-test-slave-rn98 krb5kdc[19902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:08:51.353892 19987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:51.340698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47340 (local address 127.0.196.254:41731)
0925 19:08:51.340894 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:51.340906 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:51.340941 (+ 35us) server_negotiation.cc:407] Connection header received
0925 19:08:51.341689 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:51.341698 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:51.341769 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:51.341935 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:51.342793 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.344277 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.345476 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.346199 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.346339 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:51.349585 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:51.349590 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:51.349591 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:08:51.349606 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:51.351852 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:51.352312 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:51.352317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:51.352317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:51.352526 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:51.352752 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:51.352754 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:51.352754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:51.353222 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:51.353405 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:51.353486 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 19:08:51.353649 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
I20260925 19:08:51.360755 19994 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:08:51.361311 19994 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:08:51.365342 19994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ade36a50b8b47d5a84290fccdf30ec1: Generated new TSK 0
I20260925 19:08:51.365574 19994 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.196.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
Entry for principal kudu/127.0.196.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab.
I20260925 19:08:51.398608 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:0
--local_ip_for_outbound_sockets=127.0.196.193
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:41731
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:40149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:08:51.491087 20009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:51.491320 20009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:51.491346 20009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:51.492597 20009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:51.492646 20009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:51.492705 20009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:08:51.494390 20009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:40149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/kudu_127.0.196.193.keytab
--principal=kudu/127.0.196.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:41731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:51.494699 20009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:51.494961 20009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:08:51.500422 20009 server_base.cc:1034] running on GCE node
W20260925 19:08:51.500411 20015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:51.500391 20014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:51.500391 20017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:51.500911 20009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:51.501132 20009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:51.502275 20009 hybrid_clock.cc:648] HybridClock initialized: now 1790363331502268 us; error 44 us; skew 500 ppm
Sep 25 19:08:51 dist-test-slave-rn98 krb5kdc[19902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:08:51.505309 20009 init.cc:377] Logged in from keytab as kudu/127.0.196.193@KRBTEST.COM (short username kudu)
I20260925 19:08:51.506016 20009 webserver.cc:460] Webserver started at http://127.0.196.193:34289/ using document root <none> and password file <none>
I20260925 19:08:51.506269 20009 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:51.506340 20009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:51.506486 20009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:51.507694 20009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-0/data/instance:
uuid: "eeeba54c013d4fae878a3d377521fde5"
format_stamp: "Formatted at 2026-09-25 19:08:51 on dist-test-slave-rn98"
server_key: "e8fc8fd94e8c3edf26a88b67fd0c8660"
server_key_iv: "f16aa7114faf3517040e2fa022a00fbb"
server_key_version: "encryptionkey@0"
I20260925 19:08:51.508103 20009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-0/wal/instance:
uuid: "eeeba54c013d4fae878a3d377521fde5"
format_stamp: "Formatted at 2026-09-25 19:08:51 on dist-test-slave-rn98"
server_key: "e8fc8fd94e8c3edf26a88b67fd0c8660"
server_key_iv: "f16aa7114faf3517040e2fa022a00fbb"
server_key_version: "encryptionkey@0"
I20260925 19:08:51.509550 20009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:08:51.510363 20024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:51.510598 20009 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:08:51.510747 20009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "eeeba54c013d4fae878a3d377521fde5"
format_stamp: "Formatted at 2026-09-25 19:08:51 on dist-test-slave-rn98"
server_key: "e8fc8fd94e8c3edf26a88b67fd0c8660"
server_key_iv: "f16aa7114faf3517040e2fa022a00fbb"
server_key_version: "encryptionkey@0"
I20260925 19:08:51.510831 20009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:51.525079 20009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:51.527529 20009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:51.527709 20009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:51.527974 20009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:08:51.528311 20009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:08:51.528366 20009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:51.528401 20009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:08:51.528445 20009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:51.534694 20009 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:37109
I20260925 19:08:51.534732 20137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:37109 every 8 connection(s)
I20260925 19:08:51.535106 20009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
Sep 25 19:08:51 dist-test-slave-rn98 krb5kdc[19902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.193@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:08:51.544020 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 20009
I20260925 19:08:51.544170 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-0/wal/instance
I20260925 19:08:51.544472 787 external_mini_cluster.cc:1428] Setting key c2d6a5f364a614f50c82a14dd726ac4a
I20260925 19:08:51.545683 19987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:51.535948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:41003 (local address 127.0.196.254:41731)
0925 19:08:51.536025 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:51.536027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:51.536253 (+ 226us) server_negotiation.cc:407] Connection header received
0925 19:08:51.537228 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:51.537229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:51.537239 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:51.537311 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:51.537951 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.538471 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.539591 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.539870 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.539941 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:51.542233 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:51.542234 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:51.542234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:08:51.542242 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:51.543876 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:51.544493 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:51.544494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:51.544495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:51.544586 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:51.544929 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:51.544930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:51.544930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:51.545073 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:51.545150 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:51.545520 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 19:08:51.545609 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 19:08:51.546100 20140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:51.535935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41731 (local address 127.0.196.193:41003)
0925 19:08:51.536100 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:51.536144 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:51.537117 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:51.537374 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:51.537377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:51.537648 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:08:51.537860 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:51.537864 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.538568 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.538569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:51.539478 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:51.539481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.539959 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.539960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:51.540069 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:51.540604 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:08:51.540612 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:08:51.542084 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:08:51.543990 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:51.544004 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:51.544008 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:51.544407 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:51.544709 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:51.544710 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:51.544711 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:51.544883 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:51.545161 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:08:51.545169 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:08:51.545365 (+ 196us) client_negotiation.cc:769] Sending connection context
0925 19:08:51.545502 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 19:08:51.545774 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
I20260925 19:08:51.546743 20138 heartbeater.cc:347] Connected to a master server at 127.0.196.254:41731
I20260925 19:08:51.546877 20138 heartbeater.cc:464] Registering TS with master...
I20260925 19:08:51.547078 20138 heartbeater.cc:511] Master 127.0.196.254:41731 requested a full tablet report, sending...
I20260925 19:08:51.547770 19947 ts_manager.cc:194] Registered new tserver with Master: eeeba54c013d4fae878a3d377521fde5 (127.0.196.193:37109)
I20260925 19:08:51.548444 19947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.193@KRBTEST.COM'} at 127.0.196.193:41003
WARNING: no policy specified for kudu/127.0.196.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
Entry for principal kudu/127.0.196.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab.
I20260925 19:08:51.574352 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:0
--local_ip_for_outbound_sockets=127.0.196.194
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:41731
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:40149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:08:51.659108 20143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:51.659381 20143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:51.659425 20143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:51.660682 20143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:51.660813 20143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:51.660867 20143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:08:51.662482 20143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:40149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/kudu_127.0.196.194.keytab
--principal=kudu/127.0.196.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:41731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:51.662837 20143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:51.663118 20143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:51.668517 20149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:51.668565 20148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:51.668632 20143 server_base.cc:1034] running on GCE node
W20260925 19:08:51.668802 20151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:51.669028 20143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:51.669307 20143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:51.670476 20143 hybrid_clock.cc:648] HybridClock initialized: now 1790363331670460 us; error 50 us; skew 500 ppm
Sep 25 19:08:51 dist-test-slave-rn98 krb5kdc[19902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:08:51.673146 20143 init.cc:377] Logged in from keytab as kudu/127.0.196.194@KRBTEST.COM (short username kudu)
I20260925 19:08:51.673692 20143 webserver.cc:460] Webserver started at http://127.0.196.194:45885/ using document root <none> and password file <none>
I20260925 19:08:51.673944 20143 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:51.674013 20143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:51.674170 20143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:51.675366 20143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-1/data/instance:
uuid: "b294269378ad4b2daed928f01b1276ef"
format_stamp: "Formatted at 2026-09-25 19:08:51 on dist-test-slave-rn98"
server_key: "789cae20b9afadb20e29af5d032c805e"
server_key_iv: "f5c4d852ee7ae031d9c3925d1d660a29"
server_key_version: "encryptionkey@0"
I20260925 19:08:51.675798 20143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-1/wal/instance:
uuid: "b294269378ad4b2daed928f01b1276ef"
format_stamp: "Formatted at 2026-09-25 19:08:51 on dist-test-slave-rn98"
server_key: "789cae20b9afadb20e29af5d032c805e"
server_key_iv: "f5c4d852ee7ae031d9c3925d1d660a29"
server_key_version: "encryptionkey@0"
I20260925 19:08:51.677279 20143 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:08:51.678104 20158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:51.678287 20143 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:08:51.678397 20143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "b294269378ad4b2daed928f01b1276ef"
format_stamp: "Formatted at 2026-09-25 19:08:51 on dist-test-slave-rn98"
server_key: "789cae20b9afadb20e29af5d032c805e"
server_key_iv: "f5c4d852ee7ae031d9c3925d1d660a29"
server_key_version: "encryptionkey@0"
I20260925 19:08:51.678480 20143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:51.687899 20143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:51.690413 20143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:51.690606 20143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:51.690936 20143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:08:51.691272 20143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:08:51.691334 20143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:51.691370 20143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:08:51.691418 20143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:51.697919 20143 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:37359
I20260925 19:08:51.697964 20271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:37359 every 8 connection(s)
I20260925 19:08:51.698323 20143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
I20260925 19:08:51.699798 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 20143
I20260925 19:08:51.699922 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-1/wal/instance
I20260925 19:08:51.700110 787 external_mini_cluster.cc:1428] Setting key 52b6840a93858798240385772906aa74
Sep 25 19:08:51 dist-test-slave-rn98 krb5kdc[19902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.194@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.196.195@KRBTEST.COM; defaulting to no policy
I20260925 19:08:51.710263 19987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:51.699159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:56439 (local address 127.0.196.254:41731)
0925 19:08:51.699248 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:51.699251 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:51.699457 (+ 206us) server_negotiation.cc:407] Connection header received
0925 19:08:51.700568 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:51.700569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:51.700578 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:51.700619 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:51.701356 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.702146 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.703810 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.704184 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.704266 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:51.706977 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:51.706979 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:51.706980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:08:51.706989 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:51.708769 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:51.709247 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:51.709248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:51.709249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:51.709312 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:51.709582 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:51.709583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:51.709584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:51.709751 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:51.709825 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:51.710107 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 19:08:51.710182 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 19:08:51.710460 20274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:51.699109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41731 (local address 127.0.196.194:56439)
0925 19:08:51.699324 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:51.699364 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:51.700445 (+ 1081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:51.700708 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:51.700713 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:51.701032 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:08:51.701253 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:51.701259 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.702266 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.702268 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:51.703714 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:51.703717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.704282 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.704283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:51.704408 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:51.705020 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:08:51.705029 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:08:51.706866 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:08:51.708873 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:51.708880 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:51.708883 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:51.709149 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:51.709388 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:51.709389 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:51.709389 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:51.709506 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:51.709815 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:08:51.709822 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:08:51.709961 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 19:08:51.710059 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 19:08:51.710192 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":111,"threads_started":1}
I20260925 19:08:51.711088 20272 heartbeater.cc:347] Connected to a master server at 127.0.196.254:41731
I20260925 19:08:51.711190 20272 heartbeater.cc:464] Registering TS with master...
I20260925 19:08:51.711361 20272 heartbeater.cc:511] Master 127.0.196.254:41731 requested a full tablet report, sending...
I20260925 19:08:51.711787 19947 ts_manager.cc:194] Registered new tserver with Master: b294269378ad4b2daed928f01b1276ef (127.0.196.194:37359)
I20260925 19:08:51.712152 19947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.194@KRBTEST.COM'} at 127.0.196.194:56439
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
Entry for principal kudu/127.0.196.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab.
I20260925 19:08:51.730575 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:0
--local_ip_for_outbound_sockets=127.0.196.195
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:41731
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:40149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:08:51.818224 20277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:51.818579 20277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:51.818707 20277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:51.820230 20277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:51.820331 20277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:51.820391 20277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:08:51.822458 20277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:40149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/krb5kdc/kudu_127.0.196.195.keytab
--principal=kudu/127.0.196.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:41731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:51.822923 20277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:51.823235 20277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:51.828758 20282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:51.828758 20283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:51.828831 20285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:51.829030 20277 server_base.cc:1034] running on GCE node
I20260925 19:08:51.829257 20277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:51.829560 20277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:51.830719 20277 hybrid_clock.cc:648] HybridClock initialized: now 1790363331830700 us; error 28 us; skew 500 ppm
Sep 25 19:08:51 dist-test-slave-rn98 krb5kdc[19902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:08:51.833433 20277 init.cc:377] Logged in from keytab as kudu/127.0.196.195@KRBTEST.COM (short username kudu)
I20260925 19:08:51.833954 20277 webserver.cc:460] Webserver started at http://127.0.196.195:45363/ using document root <none> and password file <none>
I20260925 19:08:51.834223 20277 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:51.834295 20277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:51.834442 20277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:51.835700 20277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-2/data/instance:
uuid: "99285d0658e346d7b954d0b9a43fce93"
format_stamp: "Formatted at 2026-09-25 19:08:51 on dist-test-slave-rn98"
server_key: "3f0e22765dde83f4ca9b88f03b4338f5"
server_key_iv: "b0561431688f1d2aaba78d4f66b56685"
server_key_version: "encryptionkey@0"
I20260925 19:08:51.836097 20277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-2/wal/instance:
uuid: "99285d0658e346d7b954d0b9a43fce93"
format_stamp: "Formatted at 2026-09-25 19:08:51 on dist-test-slave-rn98"
server_key: "3f0e22765dde83f4ca9b88f03b4338f5"
server_key_iv: "b0561431688f1d2aaba78d4f66b56685"
server_key_version: "encryptionkey@0"
I20260925 19:08:51.837545 20277 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:08:51.838380 20292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:51.838765 20277 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:08:51.838882 20277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "99285d0658e346d7b954d0b9a43fce93"
format_stamp: "Formatted at 2026-09-25 19:08:51 on dist-test-slave-rn98"
server_key: "3f0e22765dde83f4ca9b88f03b4338f5"
server_key_iv: "b0561431688f1d2aaba78d4f66b56685"
server_key_version: "encryptionkey@0"
I20260925 19:08:51.838963 20277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:51.849944 20277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:51.852995 20277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:51.853180 20277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:51.853449 20277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:08:51.853801 20277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:08:51.853857 20277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:51.853901 20277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:08:51.853932 20277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:51.860076 20277 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:35109
I20260925 19:08:51.860116 20405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:35109 every 8 connection(s)
I20260925 19:08:51.860458 20277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
I20260925 19:08:51.865978 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 20277
I20260925 19:08:51.866062 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-2/wal/instance
I20260925 19:08:51.866245 787 external_mini_cluster.cc:1428] Setting key 1524085c77f4a9dee0b1a2da116912df
Sep 25 19:08:51 dist-test-slave-rn98 krb5kdc[19902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.195@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:08:51.870254 19987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:51.861253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:53089 (local address 127.0.196.254:41731)
0925 19:08:51.861320 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:51.861322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:51.861548 (+ 226us) server_negotiation.cc:407] Connection header received
0925 19:08:51.862490 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:51.862491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:51.862502 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:51.862560 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:51.863277 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.863796 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.864799 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.865003 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.865061 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:51.867533 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:51.867534 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:51.867534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:08:51.867541 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:51.868825 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:51.869252 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:51.869253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:51.869253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:51.869304 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:51.869578 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:51.869579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:51.869579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:51.869716 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:51.869768 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:51.870104 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 19:08:51.870185 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 19:08:51.870405 20408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:51.861241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:41731 (local address 127.0.196.195:53089)
0925 19:08:51.861422 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:51.861460 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:51.862382 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:51.862681 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:51.862686 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:51.862983 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:08:51.863197 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:51.863202 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.863873 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.863873 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:51.864696 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:51.864698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.865080 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.865081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:51.865186 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:51.865805 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:08:51.865814 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:08:51.867436 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:08:51.868905 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:51.868913 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:51.868915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:51.869180 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:51.869380 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:08:51.869382 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:08:51.869382 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:08:51.869508 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:08:51.869791 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:08:51.869797 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:08:51.869959 (+ 162us) client_negotiation.cc:769] Sending connection context
0925 19:08:51.870062 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 19:08:51.870222 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20260925 19:08:51.870985 20406 heartbeater.cc:347] Connected to a master server at 127.0.196.254:41731
I20260925 19:08:51.871106 20406 heartbeater.cc:464] Registering TS with master...
I20260925 19:08:51.871320 20406 heartbeater.cc:511] Master 127.0.196.254:41731 requested a full tablet report, sending...
I20260925 19:08:51.871722 19947 ts_manager.cc:194] Registered new tserver with Master: 99285d0658e346d7b954d0b9a43fce93 (127.0.196.195:35109)
I20260925 19:08:51.872112 19947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.195@KRBTEST.COM'} at 127.0.196.195:53089
I20260925 19:08:51.878279 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:08:51.885990 19987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:51.879688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47348 (local address 127.0.196.254:41731)
0925 19:08:51.879815 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:51.879817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:51.879934 (+ 117us) server_negotiation.cc:407] Connection header received
0925 19:08:51.880024 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:51.880025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:51.880037 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:51.880106 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:51.880713 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.881247 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.882073 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.882279 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.882357 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:51.883604 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:51.883606 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:51.883606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:08:51.883614 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:51.884792 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:51.885149 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:51.885151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:51.885151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:51.885211 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:51.885405 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:51.885406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:51.885406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:51.885570 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:51.885679 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:51.885794 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 19:08:51.885881 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 19:08:51.887758 19947 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:08:51.889426 19947 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:08:51.896900 20422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:51.892498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46492 (local address 127.0.196.194:37359)
0925 19:08:51.892663 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:51.892668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:51.892981 (+ 313us) server_negotiation.cc:407] Connection header received
0925 19:08:51.893021 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:51.893023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:51.893045 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:51.893129 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:51.893231 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.893919 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.895646 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.896450 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.896533 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:51.896603 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:51.896756 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 19:08:51.896803 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
I20260925 19:08:51.896903 20417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:51.892040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:35109 (local address 127.0.0.1:49472)
0925 19:08:51.892337 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:51.892393 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:51.892482 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:51.892886 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:51.892887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:51.892889 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:51.892953 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:51.892955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.894276 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.894277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:51.895538 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:51.895541 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.896548 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.896549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:51.896640 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:51.896650 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 19:08:51.896725 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 19:08:51.896807 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":6,"thread_start_us":68,"threads_started":1}
I20260925 19:08:51.896912 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:51.892261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:37359 (local address 127.0.0.1:46492)
0925 19:08:51.892865 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:51.892876 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:51.892923 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:51.893137 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:51.893138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:51.893140 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:51.893179 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:51.893180 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.894033 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.894040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:51.895539 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:51.895541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.896531 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.896532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:51.896626 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:51.896645 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 19:08:51.896729 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 19:08:51.896817 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":72,"thread_start_us":70,"threads_started":1}
I20260925 19:08:51.897732 20421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:51.892043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:37109 (local address 127.0.0.1:53582)
0925 19:08:51.892485 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:51.892495 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:51.892545 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:51.892766 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:51.892769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:51.892772 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:51.892863 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:51.892864 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.895650 (+ 2786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.895651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:51.896348 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:51.896350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.897451 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.897452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:51.897541 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:51.897545 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:08:51.897595 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 19:08:51.897640 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":101,"threads_started":1}
I20260925 19:08:51.897903 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:51.892137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49472 (local address 127.0.196.195:35109)
0925 19:08:51.892330 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:51.892336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:51.892478 (+ 142us) server_negotiation.cc:407] Connection header received
0925 19:08:51.892736 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:51.892739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:51.892759 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:51.893034 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:51.893118 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.894183 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.895643 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.896450 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.897586 (+ 1136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:51.897714 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:51.897743 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 19:08:51.897798 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
I20260925 19:08:51.898032 20420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:51.892170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53582 (local address 127.0.196.193:37109)
0925 19:08:51.892355 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:51.892358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:51.892545 (+ 187us) server_negotiation.cc:407] Connection header received
0925 19:08:51.892663 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:51.892667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:51.892685 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:51.892756 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:51.892972 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.894178 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.896446 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.897365 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.897438 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:51.897498 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:51.897902 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 19:08:51.897954 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":74,"threads_started":1}
I20260925 19:08:51.898155 20206 tablet_service.cc:1467] Processing CreateTablet for tablet e20e8925fa9044968ff88a1426c43ac9 (DEFAULT_TABLE table=test-table [id=1e3fe5f1b3c94f579287ce70350ed0e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:08:51.898557 20206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e20e8925fa9044968ff88a1426c43ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:51.898707 20072 tablet_service.cc:1467] Processing CreateTablet for tablet e20e8925fa9044968ff88a1426c43ac9 (DEFAULT_TABLE table=test-table [id=1e3fe5f1b3c94f579287ce70350ed0e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:08:51.898708 20340 tablet_service.cc:1467] Processing CreateTablet for tablet e20e8925fa9044968ff88a1426c43ac9 (DEFAULT_TABLE table=test-table [id=1e3fe5f1b3c94f579287ce70350ed0e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:08:51.899063 20072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e20e8925fa9044968ff88a1426c43ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:51.899063 20340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e20e8925fa9044968ff88a1426c43ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:51.901682 20423 tablet_bootstrap.cc:492] T e20e8925fa9044968ff88a1426c43ac9 P b294269378ad4b2daed928f01b1276ef: Bootstrap starting.
I20260925 19:08:51.902432 20424 tablet_bootstrap.cc:492] T e20e8925fa9044968ff88a1426c43ac9 P 99285d0658e346d7b954d0b9a43fce93: Bootstrap starting.
I20260925 19:08:51.902655 20423 tablet_bootstrap.cc:654] T e20e8925fa9044968ff88a1426c43ac9 P b294269378ad4b2daed928f01b1276ef: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:51.902784 20425 tablet_bootstrap.cc:492] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5: Bootstrap starting.
I20260925 19:08:51.903206 20424 tablet_bootstrap.cc:654] T e20e8925fa9044968ff88a1426c43ac9 P 99285d0658e346d7b954d0b9a43fce93: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:51.903316 20423 log.cc:824] T e20e8925fa9044968ff88a1426c43ac9 P b294269378ad4b2daed928f01b1276ef: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:51.903393 20425 tablet_bootstrap.cc:654] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:51.903826 20424 log.cc:824] T e20e8925fa9044968ff88a1426c43ac9 P 99285d0658e346d7b954d0b9a43fce93: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:51.904024 20425 log.cc:824] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:51.904366 20423 tablet_bootstrap.cc:492] T e20e8925fa9044968ff88a1426c43ac9 P b294269378ad4b2daed928f01b1276ef: No bootstrap required, opened a new log
I20260925 19:08:51.904445 20423 ts_tablet_manager.cc:1397] T e20e8925fa9044968ff88a1426c43ac9 P b294269378ad4b2daed928f01b1276ef: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 19:08:51.904583 20424 tablet_bootstrap.cc:492] T e20e8925fa9044968ff88a1426c43ac9 P 99285d0658e346d7b954d0b9a43fce93: No bootstrap required, opened a new log
I20260925 19:08:51.904657 20424 ts_tablet_manager.cc:1397] T e20e8925fa9044968ff88a1426c43ac9 P 99285d0658e346d7b954d0b9a43fce93: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:08:51.906587 20424 raft_consensus.cc:348] T e20e8925fa9044968ff88a1426c43ac9 P 99285d0658e346d7b954d0b9a43fce93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeeba54c013d4fae878a3d377521fde5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 37109 } } peers { permanent_uuid: "99285d0658e346d7b954d0b9a43fce93" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35109 } } peers { permanent_uuid: "b294269378ad4b2daed928f01b1276ef" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 37359 } }
I20260925 19:08:51.906569 20423 raft_consensus.cc:348] T e20e8925fa9044968ff88a1426c43ac9 P b294269378ad4b2daed928f01b1276ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeeba54c013d4fae878a3d377521fde5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 37109 } } peers { permanent_uuid: "99285d0658e346d7b954d0b9a43fce93" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35109 } } peers { permanent_uuid: "b294269378ad4b2daed928f01b1276ef" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 37359 } }
I20260925 19:08:51.906806 20424 raft_consensus.cc:374] T e20e8925fa9044968ff88a1426c43ac9 P 99285d0658e346d7b954d0b9a43fce93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:51.906806 20423 raft_consensus.cc:374] T e20e8925fa9044968ff88a1426c43ac9 P b294269378ad4b2daed928f01b1276ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:51.906844 20424 raft_consensus.cc:729] T e20e8925fa9044968ff88a1426c43ac9 P 99285d0658e346d7b954d0b9a43fce93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99285d0658e346d7b954d0b9a43fce93, State: Initialized, Role: FOLLOWER
I20260925 19:08:51.906844 20423 raft_consensus.cc:729] T e20e8925fa9044968ff88a1426c43ac9 P b294269378ad4b2daed928f01b1276ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b294269378ad4b2daed928f01b1276ef, State: Initialized, Role: FOLLOWER
I20260925 19:08:51.906999 20423 consensus_queue.cc:260] T e20e8925fa9044968ff88a1426c43ac9 P b294269378ad4b2daed928f01b1276ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeeba54c013d4fae878a3d377521fde5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 37109 } } peers { permanent_uuid: "99285d0658e346d7b954d0b9a43fce93" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35109 } } peers { permanent_uuid: "b294269378ad4b2daed928f01b1276ef" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 37359 } }
I20260925 19:08:51.906996 20424 consensus_queue.cc:260] T e20e8925fa9044968ff88a1426c43ac9 P 99285d0658e346d7b954d0b9a43fce93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeeba54c013d4fae878a3d377521fde5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 37109 } } peers { permanent_uuid: "99285d0658e346d7b954d0b9a43fce93" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35109 } } peers { permanent_uuid: "b294269378ad4b2daed928f01b1276ef" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 37359 } }
I20260925 19:08:51.907312 20423 ts_tablet_manager.cc:1428] T e20e8925fa9044968ff88a1426c43ac9 P b294269378ad4b2daed928f01b1276ef: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:08:51.907312 20424 ts_tablet_manager.cc:1428] T e20e8925fa9044968ff88a1426c43ac9 P 99285d0658e346d7b954d0b9a43fce93: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260925 19:08:51.907371 20425 tablet_bootstrap.cc:492] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5: No bootstrap required, opened a new log
I20260925 19:08:51.907454 20425 ts_tablet_manager.cc:1397] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 19:08:51.907469 20406 heartbeater.cc:503] Master 127.0.196.254:41731 was elected leader, sending a full tablet report...
I20260925 19:08:51.909292 20425 raft_consensus.cc:348] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeeba54c013d4fae878a3d377521fde5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 37109 } } peers { permanent_uuid: "99285d0658e346d7b954d0b9a43fce93" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35109 } } peers { permanent_uuid: "b294269378ad4b2daed928f01b1276ef" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 37359 } }
I20260925 19:08:51.909446 20425 raft_consensus.cc:374] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:51.909468 20425 raft_consensus.cc:729] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeeba54c013d4fae878a3d377521fde5, State: Initialized, Role: FOLLOWER
I20260925 19:08:51.909588 20425 consensus_queue.cc:260] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeeba54c013d4fae878a3d377521fde5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 37109 } } peers { permanent_uuid: "99285d0658e346d7b954d0b9a43fce93" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35109 } } peers { permanent_uuid: "b294269378ad4b2daed928f01b1276ef" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 37359 } }
I20260925 19:08:51.909893 20425 ts_tablet_manager.cc:1428] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:08:51.909917 20138 heartbeater.cc:503] Master 127.0.196.254:41731 was elected leader, sending a full tablet report...
I20260925 19:08:51.910043 20272 heartbeater.cc:503] Master 127.0.196.254:41731 was elected leader, sending a full tablet report...
I20260925 19:08:51.941494 20431 raft_consensus.cc:482] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:08:51.941671 20431 raft_consensus.cc:504] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeeba54c013d4fae878a3d377521fde5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 37109 } } peers { permanent_uuid: "99285d0658e346d7b954d0b9a43fce93" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35109 } } peers { permanent_uuid: "b294269378ad4b2daed928f01b1276ef" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 37359 } }
I20260925 19:08:51.942060 20431 leader_election.cc:288] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99285d0658e346d7b954d0b9a43fce93 (127.0.196.195:35109), b294269378ad4b2daed928f01b1276ef (127.0.196.194:37359)
I20260925 19:08:51.945398 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:51.942370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:45655 (local address 127.0.196.195:35109)
0925 19:08:51.942439 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:51.942441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:51.942487 (+ 46us) server_negotiation.cc:407] Connection header received
0925 19:08:51.942582 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:51.942583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:51.942594 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:51.942682 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:51.942936 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.943406 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.944505 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.945003 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.945086 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:51.945097 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:51.945269 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 19:08:51.945310 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 19:08:51.945459 20422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:51.942561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:54643 (local address 127.0.196.194:37359)
0925 19:08:51.942659 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:51.942662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:51.942804 (+ 142us) server_negotiation.cc:407] Connection header received
0925 19:08:51.942871 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:51.942872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:51.942901 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:51.942952 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:51.943105 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.943574 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.944608 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.945086 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.945177 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:51.945186 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:51.945345 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 19:08:51.945398 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 19:08:51.945468 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:51.942451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:37359 (local address 127.0.196.193:54643)
0925 19:08:51.942711 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:51.942748 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:51.942816 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:51.942957 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:51.942958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:51.942959 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:51.943031 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:51.943033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.943675 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.943676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:51.944497 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:51.944500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.945175 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.945176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:51.945253 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:51.945263 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 19:08:51.945338 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 19:08:51.945382 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":222,"threads_started":1}
I20260925 19:08:51.945538 20140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:51.942310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:35109 (local address 127.0.196.193:45655)
0925 19:08:51.942418 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:51.942433 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:51.942508 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:51.942741 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:51.942742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:51.942744 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:51.942828 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:51.942830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.943495 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.943496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:51.944390 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:51.944393 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.945103 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.945105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:51.945187 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:51.945190 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:08:51.945264 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 19:08:51.945310 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 19:08:51.945900 20360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e20e8925fa9044968ff88a1426c43ac9" candidate_uuid: "eeeba54c013d4fae878a3d377521fde5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99285d0658e346d7b954d0b9a43fce93" is_pre_election: true
I20260925 19:08:51.946075 20360 raft_consensus.cc:2454] T e20e8925fa9044968ff88a1426c43ac9 P 99285d0658e346d7b954d0b9a43fce93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeeba54c013d4fae878a3d377521fde5 in term 0.
I20260925 19:08:51.946152 20226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e20e8925fa9044968ff88a1426c43ac9" candidate_uuid: "eeeba54c013d4fae878a3d377521fde5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b294269378ad4b2daed928f01b1276ef" is_pre_election: true
I20260925 19:08:51.946306 20226 raft_consensus.cc:2454] T e20e8925fa9044968ff88a1426c43ac9 P b294269378ad4b2daed928f01b1276ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeeba54c013d4fae878a3d377521fde5 in term 0.
I20260925 19:08:51.946339 20027 leader_election.cc:302] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99285d0658e346d7b954d0b9a43fce93, eeeba54c013d4fae878a3d377521fde5; no voters:
I20260925 19:08:51.946491 20431 raft_consensus.cc:2781] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:08:51.946549 20431 raft_consensus.cc:482] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:08:51.946583 20431 raft_consensus.cc:3037] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:51.947342 20431 raft_consensus.cc:504] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeeba54c013d4fae878a3d377521fde5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 37109 } } peers { permanent_uuid: "99285d0658e346d7b954d0b9a43fce93" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35109 } } peers { permanent_uuid: "b294269378ad4b2daed928f01b1276ef" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 37359 } }
I20260925 19:08:51.947496 20431 leader_election.cc:288] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5 [CANDIDATE]: Term 1 election: Requested vote from peers 99285d0658e346d7b954d0b9a43fce93 (127.0.196.195:35109), b294269378ad4b2daed928f01b1276ef (127.0.196.194:37359)
I20260925 19:08:51.947706 20360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e20e8925fa9044968ff88a1426c43ac9" candidate_uuid: "eeeba54c013d4fae878a3d377521fde5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99285d0658e346d7b954d0b9a43fce93"
I20260925 19:08:51.947734 20226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e20e8925fa9044968ff88a1426c43ac9" candidate_uuid: "eeeba54c013d4fae878a3d377521fde5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b294269378ad4b2daed928f01b1276ef"
I20260925 19:08:51.947808 20360 raft_consensus.cc:3037] T e20e8925fa9044968ff88a1426c43ac9 P 99285d0658e346d7b954d0b9a43fce93 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:51.947808 20226 raft_consensus.cc:3037] T e20e8925fa9044968ff88a1426c43ac9 P b294269378ad4b2daed928f01b1276ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:51.948803 20360 raft_consensus.cc:2454] T e20e8925fa9044968ff88a1426c43ac9 P 99285d0658e346d7b954d0b9a43fce93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeeba54c013d4fae878a3d377521fde5 in term 1.
I20260925 19:08:51.948803 20226 raft_consensus.cc:2454] T e20e8925fa9044968ff88a1426c43ac9 P b294269378ad4b2daed928f01b1276ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeeba54c013d4fae878a3d377521fde5 in term 1.
I20260925 19:08:51.949031 20027 leader_election.cc:302] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b294269378ad4b2daed928f01b1276ef, eeeba54c013d4fae878a3d377521fde5; no voters:
W20260925 19:08:51.949039 20273 tablet.cc:2369] T e20e8925fa9044968ff88a1426c43ac9 P b294269378ad4b2daed928f01b1276ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:08:51.949146 20431 raft_consensus.cc:2781] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:08:51.949307 20431 raft_consensus.cc:686] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5 [term 1 LEADER]: Becoming Leader. State: Replica: eeeba54c013d4fae878a3d377521fde5, State: Running, Role: LEADER
I20260925 19:08:51.949445 20431 consensus_queue.cc:237] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeeba54c013d4fae878a3d377521fde5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 37109 } } peers { permanent_uuid: "99285d0658e346d7b954d0b9a43fce93" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35109 } } peers { permanent_uuid: "b294269378ad4b2daed928f01b1276ef" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 37359 } }
I20260925 19:08:51.950407 19946 catalog_manager.cc:5495] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5 reported cstate change: term changed from 0 to 1, leader changed from <none> to eeeba54c013d4fae878a3d377521fde5 (127.0.196.193). New cstate: current_term: 1 leader_uuid: "eeeba54c013d4fae878a3d377521fde5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeeba54c013d4fae878a3d377521fde5" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 37109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99285d0658e346d7b954d0b9a43fce93" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b294269378ad4b2daed928f01b1276ef" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 37359 } health_report { overall_health: UNKNOWN } } }
I20260925 19:08:51.971236 20420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:51.968503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53584 (local address 127.0.196.193:37109)
0925 19:08:51.968585 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:51.968587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:51.968596 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:08:51.968645 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:51.968646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:51.968665 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:51.968789 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:08:51.968933 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.969409 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.970251 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:51.970451 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:51.970578 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:51.970833 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:08:51.970950 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:08:51.971062 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:51.971081 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 19:08:51.971107 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 19:08:51.973385 20226 raft_consensus.cc:1261] T e20e8925fa9044968ff88a1426c43ac9 P b294269378ad4b2daed928f01b1276ef [term 1 FOLLOWER]: Refusing update from remote peer eeeba54c013d4fae878a3d377521fde5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:08:51.973385 20360 raft_consensus.cc:1261] T e20e8925fa9044968ff88a1426c43ac9 P 99285d0658e346d7b954d0b9a43fce93 [term 1 FOLLOWER]: Refusing update from remote peer eeeba54c013d4fae878a3d377521fde5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:08:51.973853 20433 consensus_queue.cc:1035] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99285d0658e346d7b954d0b9a43fce93" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 35109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:08:51.974015 20431 consensus_queue.cc:1035] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b294269378ad4b2daed928f01b1276ef" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:08:51.976575 20436 mvcc.cc:205] Tried to move back new op lower bound from 7333328207759413248 to 7333328207665799168. Current Snapshot: MvccSnapshot[applied={T|T < 7333328207759413248}]
I20260925 19:08:51.977280 20437 mvcc.cc:205] Tried to move back new op lower bound from 7333328207759413248 to 7333328207665799168. Current Snapshot: MvccSnapshot[applied={T|T < 7333328207759413248}]
I20260925 19:08:51.977365 20438 mvcc.cc:205] Tried to move back new op lower bound from 7333328207759413248 to 7333328207665799168. Current Snapshot: MvccSnapshot[applied={T|T < 7333328207759413248}]
I20260925 19:08:51.979457 19947 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:08:51.979597 19947 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:08:51.980450 19947 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ade36a50b8b47d5a84290fccdf30ec1: Sending DeleteTablet for 3 replicas of tablet e20e8925fa9044968ff88a1426c43ac9
I20260925 19:08:51.980851 20340 tablet_service.cc:1514] Processing DeleteTablet for tablet e20e8925fa9044968ff88a1426c43ac9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:08:51 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:49472
I20260925 19:08:51.980880 20072 tablet_service.cc:1514] Processing DeleteTablet for tablet e20e8925fa9044968ff88a1426c43ac9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:08:51 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:53582
I20260925 19:08:51.981125 20206 tablet_service.cc:1514] Processing DeleteTablet for tablet e20e8925fa9044968ff88a1426c43ac9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:08:51 UTC) from {username='kudu', principal='kudu/127.0.196.254@KRBTEST.COM'} at 127.0.0.1:46492
I20260925 19:08:51.981290 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 20009
I20260925 19:08:51.981309 20445 tablet_replica.cc:330] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5: stopping tablet replica
I20260925 19:08:51.981410 20445 raft_consensus.cc:2229] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:08:51.981515 20445 raft_consensus.cc:2258] T e20e8925fa9044968ff88a1426c43ac9 P eeeba54c013d4fae878a3d377521fde5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:08:51.981160 20446 tablet_replica.cc:330] T e20e8925fa9044968ff88a1426c43ac9 P 99285d0658e346d7b954d0b9a43fce93: stopping tablet replica
I20260925 19:08:51.981686 20446 raft_consensus.cc:2229] T e20e8925fa9044968ff88a1426c43ac9 P 99285d0658e346d7b954d0b9a43fce93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:08:51.981873 20446 raft_consensus.cc:2258] T e20e8925fa9044968ff88a1426c43ac9 P 99285d0658e346d7b954d0b9a43fce93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:08:51.982366 20447 tablet_replica.cc:330] T e20e8925fa9044968ff88a1426c43ac9 P b294269378ad4b2daed928f01b1276ef: stopping tablet replica
I20260925 19:08:51.982503 20447 raft_consensus.cc:2229] T e20e8925fa9044968ff88a1426c43ac9 P b294269378ad4b2daed928f01b1276ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:08:51.982560 20446 ts_tablet_manager.cc:1905] T e20e8925fa9044968ff88a1426c43ac9 P 99285d0658e346d7b954d0b9a43fce93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:08:51.982678 20447 raft_consensus.cc:2258] T e20e8925fa9044968ff88a1426c43ac9 P b294269378ad4b2daed928f01b1276ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:08:51.983074 20447 ts_tablet_manager.cc:1905] T e20e8925fa9044968ff88a1426c43ac9 P b294269378ad4b2daed928f01b1276ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:08:51.984687 20446 ts_tablet_manager.cc:1918] T e20e8925fa9044968ff88a1426c43ac9 P 99285d0658e346d7b954d0b9a43fce93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:08:51.984820 20446 log.cc:1196] T e20e8925fa9044968ff88a1426c43ac9 P 99285d0658e346d7b954d0b9a43fce93: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-2/wal/wals/e20e8925fa9044968ff88a1426c43ac9
I20260925 19:08:51.984855 20447 ts_tablet_manager.cc:1918] T e20e8925fa9044968ff88a1426c43ac9 P b294269378ad4b2daed928f01b1276ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:08:51.984928 20447 log.cc:1196] T e20e8925fa9044968ff88a1426c43ac9 P b294269378ad4b2daed928f01b1276ef: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790363087086945-787-0/minicluster-data/ts-1/wal/wals/e20e8925fa9044968ff88a1426c43ac9
I20260925 19:08:51.985123 20446 ts_tablet_manager.cc:1939] T e20e8925fa9044968ff88a1426c43ac9 P 99285d0658e346d7b954d0b9a43fce93: Deleting consensus metadata
I20260925 19:08:51.985251 20447 ts_tablet_manager.cc:1939] T e20e8925fa9044968ff88a1426c43ac9 P b294269378ad4b2daed928f01b1276ef: Deleting consensus metadata
I20260925 19:08:51.985736 19933 catalog_manager.cc:4841] TS 99285d0658e346d7b954d0b9a43fce93 (127.0.196.195:35109): tablet e20e8925fa9044968ff88a1426c43ac9 (table test-table [id=1e3fe5f1b3c94f579287ce70350ed0e6]) successfully deleted
I20260925 19:08:51.985899 19933 catalog_manager.cc:4841] TS b294269378ad4b2daed928f01b1276ef (127.0.196.194:37359): tablet e20e8925fa9044968ff88a1426c43ac9 (table test-table [id=1e3fe5f1b3c94f579287ce70350ed0e6]) successfully deleted
W20260925 19:08:51.988561 19934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.196.193:37109 (error 108)
I20260925 19:08:51.989132 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:51.988987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:37109 (local address 127.0.0.1:53588)
0925 19:08:51.989053 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:51.989094 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.193:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 19:08:51.989213 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 20143
W20260925 19:08:51.989296 19934 catalog_manager.cc:4584] TS eeeba54c013d4fae878a3d377521fde5 (127.0.196.193:37109): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e20e8925fa9044968ff88a1426c43ac9: Network error: Client connection negotiation failed: client connection to 127.0.196.193:37109: connect: Connection refused (error 111)
I20260925 19:08:51.994493 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 20277
I20260925 19:08:51.999414 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 19915
2026-09-25T19:08:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1973 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T19:08:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:08:52Z Disabled control of system clock
I20260925 19:08:52.031025 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:38399
--webserver_interface=127.0.196.254
--webserver_port=0
--builtin_ntp_servers=127.0.196.212:44203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:38399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 19:08:52.118541 20452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:52.118834 20452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:52.118857 20452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:52.120052 20452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:08:52.120095 20452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:52.120173 20452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:52.120205 20452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:08:52.120223 20452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:08:52.122929 20452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:44203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:38399
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:38399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:52.123291 20452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:52.123710 20452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:52.129882 20457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:52.129884 20458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:52.129935 20452 server_base.cc:1034] running on GCE node
W20260925 19:08:52.129891 20460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:52.130429 20452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:52.130824 20452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:52.131987 20452 hybrid_clock.cc:648] HybridClock initialized: now 1790363332131953 us; error 50 us; skew 500 ppm
I20260925 19:08:52.133422 20452 webserver.cc:460] Webserver started at http://127.0.196.254:40183/ using document root <none> and password file <none>
I20260925 19:08:52.133750 20452 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:52.133838 20452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:52.133998 20452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:52.135543 20452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "47b42efdbb194ce891495863b92d1417"
format_stamp: "Formatted at 2026-09-25 19:08:52 on dist-test-slave-rn98"
server_key: "543ff79d02befecd8eb3c0b7e27a5f01"
server_key_iv: "cfdd30f1ba93fc3dfbc23c597e764ed6"
server_key_version: "encryptionkey@0"
I20260925 19:08:52.136000 20452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "47b42efdbb194ce891495863b92d1417"
format_stamp: "Formatted at 2026-09-25 19:08:52 on dist-test-slave-rn98"
server_key: "543ff79d02befecd8eb3c0b7e27a5f01"
server_key_iv: "cfdd30f1ba93fc3dfbc23c597e764ed6"
server_key_version: "encryptionkey@0"
I20260925 19:08:52.137609 20452 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:08:52.138561 20466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:52.138823 20452 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:08:52.138954 20452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "47b42efdbb194ce891495863b92d1417"
format_stamp: "Formatted at 2026-09-25 19:08:52 on dist-test-slave-rn98"
server_key: "543ff79d02befecd8eb3c0b7e27a5f01"
server_key_iv: "cfdd30f1ba93fc3dfbc23c597e764ed6"
server_key_version: "encryptionkey@0"
I20260925 19:08:52.139050 20452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:52.153779 20452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:52.154168 20452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:52.154351 20452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:52.158381 20452 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:38399
I20260925 19:08:52.158461 20518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:38399 every 8 connection(s)
I20260925 19:08:52.158879 20452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:08:52.160019 20519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:52.162569 20519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47b42efdbb194ce891495863b92d1417: Bootstrap starting.
I20260925 19:08:52.163341 20519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47b42efdbb194ce891495863b92d1417: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:52.163687 20519 log.cc:824] T 00000000000000000000000000000000 P 47b42efdbb194ce891495863b92d1417: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:52.164480 20519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47b42efdbb194ce891495863b92d1417: No bootstrap required, opened a new log
I20260925 19:08:52.166404 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 20452
I20260925 19:08:52.166541 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:08:52.166781 787 external_mini_cluster.cc:1428] Setting key 7e15ddb72894d4e7a499ea9dc850752b
I20260925 19:08:52.166602 20519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47b42efdbb194ce891495863b92d1417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47b42efdbb194ce891495863b92d1417" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38399 } }
I20260925 19:08:52.166831 20519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47b42efdbb194ce891495863b92d1417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:52.166869 20519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47b42efdbb194ce891495863b92d1417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47b42efdbb194ce891495863b92d1417, State: Initialized, Role: FOLLOWER
I20260925 19:08:52.167030 20519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47b42efdbb194ce891495863b92d1417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47b42efdbb194ce891495863b92d1417" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38399 } }
I20260925 19:08:52.167133 20519 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47b42efdbb194ce891495863b92d1417 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:08:52.167172 20519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47b42efdbb194ce891495863b92d1417 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:08:52.167243 20519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47b42efdbb194ce891495863b92d1417 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:52.168090 20519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47b42efdbb194ce891495863b92d1417 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47b42efdbb194ce891495863b92d1417" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38399 } }
I20260925 19:08:52.168212 20519 leader_election.cc:302] T 00000000000000000000000000000000 P 47b42efdbb194ce891495863b92d1417 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47b42efdbb194ce891495863b92d1417; no voters:
I20260925 19:08:52.168442 20519 leader_election.cc:288] T 00000000000000000000000000000000 P 47b42efdbb194ce891495863b92d1417 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:08:52.168466 20524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47b42efdbb194ce891495863b92d1417 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:08:52.168586 20524 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47b42efdbb194ce891495863b92d1417 [term 1 LEADER]: Becoming Leader. State: Replica: 47b42efdbb194ce891495863b92d1417, State: Running, Role: LEADER
I20260925 19:08:52.168672 20524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47b42efdbb194ce891495863b92d1417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47b42efdbb194ce891495863b92d1417" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38399 } }
I20260925 19:08:52.168740 20519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47b42efdbb194ce891495863b92d1417 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:08:52.169059 20526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47b42efdbb194ce891495863b92d1417 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47b42efdbb194ce891495863b92d1417. Latest consensus state: current_term: 1 leader_uuid: "47b42efdbb194ce891495863b92d1417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47b42efdbb194ce891495863b92d1417" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38399 } } }
I20260925 19:08:52.169330 20526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47b42efdbb194ce891495863b92d1417 [sys.catalog]: This master's current role is: LEADER
I20260925 19:08:52.169260 20525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47b42efdbb194ce891495863b92d1417 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47b42efdbb194ce891495863b92d1417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47b42efdbb194ce891495863b92d1417" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 38399 } } }
I20260925 19:08:52.169623 20525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47b42efdbb194ce891495863b92d1417 [sys.catalog]: This master's current role is: LEADER
I20260925 19:08:52.169827 20533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:08:52.170388 20533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:08:52.172191 20533 catalog_manager.cc:1347] Generated new cluster ID: 7df973ffebb44588af843be09396c192
I20260925 19:08:52.172262 20533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:08:52.174006 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:52.167349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53652 (local address 127.0.196.254:38399)
0925 19:08:52.167632 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:52.167643 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:52.167679 (+ 36us) server_negotiation.cc:407] Connection header received
0925 19:08:52.167818 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:52.167826 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:52.167918 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:52.168080 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:52.168408 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.170222 (+ 1814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.171425 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.172238 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.172762 (+ 524us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:52.173247 (+ 485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:52.173251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:52.173251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:08:52.173467 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:52.173514 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:08:52.173531 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:52.173608 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:52.173685 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 19:08:52.173772 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":120,"threads_started":1}
I20260925 19:08:52.187179 20533 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:08:52.187788 20533 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:08:52.193694 20533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47b42efdbb194ce891495863b92d1417: Generated new TSK 0
I20260925 19:08:52.193972 20533 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:08:52.199236 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.193:0
--local_ip_for_outbound_sockets=127.0.196.193
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:38399
--builtin_ntp_servers=127.0.196.212:44203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 19:08:52.289206 20543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:52.289503 20543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:52.289549 20543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:52.291150 20543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:52.291245 20543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:52.291298 20543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.193
I20260925 19:08:52.292984 20543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:44203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.196.193
--webserver_port=0
--tserver_master_addrs=127.0.196.254:38399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.193
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:52.293318 20543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:52.293733 20543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:52.299453 20548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:52.299489 20551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:52.299654 20543 server_base.cc:1034] running on GCE node
W20260925 19:08:52.299528 20549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:52.300071 20543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:52.300439 20543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:52.301641 20543 hybrid_clock.cc:648] HybridClock initialized: now 1790363332301617 us; error 44 us; skew 500 ppm
I20260925 19:08:52.303229 20543 webserver.cc:460] Webserver started at http://127.0.196.193:33159/ using document root <none> and password file <none>
I20260925 19:08:52.303510 20543 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:52.303568 20543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:52.303743 20543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:52.305009 20543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-0/data/instance:
uuid: "b275bde3f075438a9571bb5cb48211c4"
format_stamp: "Formatted at 2026-09-25 19:08:52 on dist-test-slave-rn98"
server_key: "011fef82472cd9adfef71119b3f88b0b"
server_key_iv: "035119abc9e438414bc3f994ab664f57"
server_key_version: "encryptionkey@0"
I20260925 19:08:52.305405 20543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-0/wal/instance:
uuid: "b275bde3f075438a9571bb5cb48211c4"
format_stamp: "Formatted at 2026-09-25 19:08:52 on dist-test-slave-rn98"
server_key: "011fef82472cd9adfef71119b3f88b0b"
server_key_iv: "035119abc9e438414bc3f994ab664f57"
server_key_version: "encryptionkey@0"
I20260925 19:08:52.306921 20543 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:08:52.307794 20557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:52.308005 20543 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:08:52.308146 20543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-0/wal
uuid: "b275bde3f075438a9571bb5cb48211c4"
format_stamp: "Formatted at 2026-09-25 19:08:52 on dist-test-slave-rn98"
server_key: "011fef82472cd9adfef71119b3f88b0b"
server_key_iv: "035119abc9e438414bc3f994ab664f57"
server_key_version: "encryptionkey@0"
I20260925 19:08:52.308249 20543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:52.336035 20543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:52.336432 20543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:52.336613 20543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:52.336910 20543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:08:52.337311 20543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:08:52.337379 20543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:52.337424 20543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:08:52.337447 20543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:52.344327 20543 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.193:33485
I20260925 19:08:52.344372 20670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.193:33485 every 8 connection(s)
I20260925 19:08:52.344828 20543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-0/data/info.pb
I20260925 19:08:52.345255 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 20543
I20260925 19:08:52.345396 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-0/wal/instance
I20260925 19:08:52.345649 787 external_mini_cluster.cc:1428] Setting key 2b35c5a86d06f387d4dd3b3399d2a121
I20260925 19:08:52.346825 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.194:0
--local_ip_for_outbound_sockets=127.0.196.194
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:38399
--builtin_ntp_servers=127.0.196.212:44203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 19:08:52.351850 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:52.345654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.193:33955 (local address 127.0.196.254:38399)
0925 19:08:52.345711 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:52.345714 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:52.346072 (+ 358us) server_negotiation.cc:407] Connection header received
0925 19:08:52.346921 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:52.346922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:52.346939 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:52.347011 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:52.347848 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.348770 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.349831 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.350212 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.350302 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:52.351407 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:52.351410 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:52.351410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:08:52.351418 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:52.351439 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:08:52.351444 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:52.351518 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:52.351648 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 19:08:52.351693 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:08:52.351904 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:52.345672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38399 (local address 127.0.196.193:33955)
0925 19:08:52.345901 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:52.345969 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:52.346792 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:52.347109 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:52.347115 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:52.347128 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:08:52.347730 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:52.347736 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.348890 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.348891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:52.349724 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:52.349727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.350338 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.350339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:52.350502 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:52.351239 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:08:52.351240 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:08:52.351274 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:08:52.351300 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:08:52.351554 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:08:52.351559 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:08:52.351562 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:08:52.351620 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 19:08:52.351715 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":103,"threads_started":1}
I20260925 19:08:52.352525 20671 heartbeater.cc:347] Connected to a master server at 127.0.196.254:38399
I20260925 19:08:52.352768 20671 heartbeater.cc:464] Registering TS with master...
I20260925 19:08:52.353039 20671 heartbeater.cc:511] Master 127.0.196.254:38399 requested a full tablet report, sending...
I20260925 19:08:52.353677 20483 ts_manager.cc:194] Registered new tserver with Master: b275bde3f075438a9571bb5cb48211c4 (127.0.196.193:33485)
I20260925 19:08:52.354379 20483 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.196.193:33955
W20260925 19:08:52.435145 20674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:52.435375 20674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:52.435400 20674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:52.436678 20674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:52.436738 20674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:52.436766 20674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.194
I20260925 19:08:52.438259 20674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:44203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.196.194
--webserver_port=0
--tserver_master_addrs=127.0.196.254:38399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.194
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:52.438551 20674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:52.438931 20674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:08:52.445303 20674 server_base.cc:1034] running on GCE node
W20260925 19:08:52.445284 20680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:52.445286 20682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:52.445282 20679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:52.445778 20674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:52.446084 20674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:52.447322 20674 hybrid_clock.cc:648] HybridClock initialized: now 1790363332447300 us; error 32 us; skew 500 ppm
I20260925 19:08:52.448979 20674 webserver.cc:460] Webserver started at http://127.0.196.194:34587/ using document root <none> and password file <none>
I20260925 19:08:52.449311 20674 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:52.449396 20674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:52.449592 20674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:52.450907 20674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-1/data/instance:
uuid: "03ca2a4171b64dc8ac94e62b3417c6d0"
format_stamp: "Formatted at 2026-09-25 19:08:52 on dist-test-slave-rn98"
server_key: "dbbcbcd797ed6e05bf7022fa7b6c4ae5"
server_key_iv: "67a84b24e000c3c0454d2bfd0f128235"
server_key_version: "encryptionkey@0"
I20260925 19:08:52.451315 20674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-1/wal/instance:
uuid: "03ca2a4171b64dc8ac94e62b3417c6d0"
format_stamp: "Formatted at 2026-09-25 19:08:52 on dist-test-slave-rn98"
server_key: "dbbcbcd797ed6e05bf7022fa7b6c4ae5"
server_key_iv: "67a84b24e000c3c0454d2bfd0f128235"
server_key_version: "encryptionkey@0"
I20260925 19:08:52.452958 20674 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:08:52.453886 20688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:52.454242 20674 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:08:52.454387 20674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-1/wal
uuid: "03ca2a4171b64dc8ac94e62b3417c6d0"
format_stamp: "Formatted at 2026-09-25 19:08:52 on dist-test-slave-rn98"
server_key: "dbbcbcd797ed6e05bf7022fa7b6c4ae5"
server_key_iv: "67a84b24e000c3c0454d2bfd0f128235"
server_key_version: "encryptionkey@0"
I20260925 19:08:52.454484 20674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:52.482412 20674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:52.482873 20674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:52.483054 20674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:52.483343 20674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:08:52.483703 20674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:08:52.483767 20674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:52.483808 20674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:08:52.483837 20674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:52.489746 20674 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.194:34471
I20260925 19:08:52.489813 20801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.194:34471 every 8 connection(s)
I20260925 19:08:52.490226 20674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-1/data/info.pb
I20260925 19:08:52.493129 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 20674
I20260925 19:08:52.493270 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-1/wal/instance
I20260925 19:08:52.493484 787 external_mini_cluster.cc:1428] Setting key f19696fdbdc7442f955a08d0514660cf
I20260925 19:08:52.494604 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.196.195:0
--local_ip_for_outbound_sockets=127.0.196.195
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:38399
--builtin_ntp_servers=127.0.196.212:44203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 19:08:52.497082 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:52.491225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.194:38319 (local address 127.0.196.254:38399)
0925 19:08:52.491308 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:52.491311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:52.491588 (+ 277us) server_negotiation.cc:407] Connection header received
0925 19:08:52.492081 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:52.492082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:52.492093 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:52.492151 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:52.492929 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.493864 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.495319 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.495670 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.495752 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:52.496721 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:52.496723 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:52.496723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:08:52.496729 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:52.496749 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:08:52.496754 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:52.496819 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:52.496926 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 19:08:52.496980 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 19:08:52.497236 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:52.491209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38399 (local address 127.0.196.194:38319)
0925 19:08:52.491448 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:52.491485 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:52.491987 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:52.492215 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:52.492219 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:52.492227 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:08:52.492741 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:52.492746 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.493948 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.493949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:52.495186 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:52.495189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.495779 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.495781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:52.495923 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:52.496549 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:08:52.496549 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:08:52.496585 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:08:52.496613 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:08:52.496828 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:08:52.496832 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:08:52.496835 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:08:52.496892 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 19:08:52.496986 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":117,"threads_started":1}
I20260925 19:08:52.497762 20802 heartbeater.cc:347] Connected to a master server at 127.0.196.254:38399
I20260925 19:08:52.497874 20802 heartbeater.cc:464] Registering TS with master...
I20260925 19:08:52.498080 20802 heartbeater.cc:511] Master 127.0.196.254:38399 requested a full tablet report, sending...
I20260925 19:08:52.498564 20483 ts_manager.cc:194] Registered new tserver with Master: 03ca2a4171b64dc8ac94e62b3417c6d0 (127.0.196.194:34471)
I20260925 19:08:52.498968 20483 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.196.194:38319
W20260925 19:08:52.580401 20805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:52.580631 20805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:52.580657 20805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:52.581981 20805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:52.582041 20805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:52.582068 20805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.195
I20260925 19:08:52.583590 20805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:44203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.196.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.196.195
--webserver_port=0
--tserver_master_addrs=127.0.196.254:38399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.196.195
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:52.583896 20805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:52.584249 20805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:52.589964 20810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:52.590148 20813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:52.590227 20805 server_base.cc:1034] running on GCE node
W20260925 19:08:52.590438 20811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:52.590912 20805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:52.591259 20805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:52.592464 20805 hybrid_clock.cc:648] HybridClock initialized: now 1790363332592452 us; error 37 us; skew 500 ppm
I20260925 19:08:52.594069 20805 webserver.cc:460] Webserver started at http://127.0.196.195:36503/ using document root <none> and password file <none>
I20260925 19:08:52.594370 20805 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:52.594455 20805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:52.594795 20805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:52.596146 20805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-2/data/instance:
uuid: "be33d00724774dc8879a3bd3db1a166e"
format_stamp: "Formatted at 2026-09-25 19:08:52 on dist-test-slave-rn98"
server_key: "cc4f41135d08af66671381f5dff185b3"
server_key_iv: "a9c5d4bbe1aa6419cc45c51945ab4463"
server_key_version: "encryptionkey@0"
I20260925 19:08:52.596540 20805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-2/wal/instance:
uuid: "be33d00724774dc8879a3bd3db1a166e"
format_stamp: "Formatted at 2026-09-25 19:08:52 on dist-test-slave-rn98"
server_key: "cc4f41135d08af66671381f5dff185b3"
server_key_iv: "a9c5d4bbe1aa6419cc45c51945ab4463"
server_key_version: "encryptionkey@0"
I20260925 19:08:52.598024 20805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:08:52.599002 20819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:52.599323 20805 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:08:52.599469 20805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-2/wal
uuid: "be33d00724774dc8879a3bd3db1a166e"
format_stamp: "Formatted at 2026-09-25 19:08:52 on dist-test-slave-rn98"
server_key: "cc4f41135d08af66671381f5dff185b3"
server_key_iv: "a9c5d4bbe1aa6419cc45c51945ab4463"
server_key_version: "encryptionkey@0"
I20260925 19:08:52.599582 20805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:52.611497 20805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:52.611907 20805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:52.612092 20805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:52.612386 20805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:08:52.612787 20805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:08:52.612857 20805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:52.612905 20805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:08:52.612928 20805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:52.619650 20805 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.195:44163
I20260925 19:08:52.619753 20932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.195:44163 every 8 connection(s)
I20260925 19:08:52.620116 20805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-2/data/info.pb
I20260925 19:08:52.621184 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 20805
I20260925 19:08:52.621305 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-2/wal/instance
I20260925 19:08:52.621497 787 external_mini_cluster.cc:1428] Setting key e6656b397722854c4d39abdff5dbaf99
I20260925 19:08:52.626158 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:52.620947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:53565 (local address 127.0.196.254:38399)
0925 19:08:52.621023 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:52.621026 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:52.621344 (+ 318us) server_negotiation.cc:407] Connection header received
0925 19:08:52.621951 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:52.621952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:52.621963 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:52.622048 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:52.623034 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.623542 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.624666 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.624870 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.624933 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:52.625892 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:52.625894 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:52.625894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:08:52.625899 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:52.625918 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:08:52.625922 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:52.625971 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:52.626062 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 19:08:52.626093 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 19:08:52.626358 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:52.620945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:38399 (local address 127.0.196.195:53565)
0925 19:08:52.621188 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:52.621234 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:52.621794 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:52.622159 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:52.622165 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:52.622176 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:08:52.622926 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:52.622933 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.623669 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.623670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:52.624545 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:52.624547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.624955 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.624956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:52.625078 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:52.625726 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:08:52.625726 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:08:52.625763 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:08:52.625787 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:08:52.625985 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:08:52.625989 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:08:52.625993 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:08:52.626048 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 19:08:52.626138 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":108,"threads_started":1}
I20260925 19:08:52.626945 20933 heartbeater.cc:347] Connected to a master server at 127.0.196.254:38399
I20260925 19:08:52.627045 20933 heartbeater.cc:464] Registering TS with master...
I20260925 19:08:52.627198 20933 heartbeater.cc:511] Master 127.0.196.254:38399 requested a full tablet report, sending...
I20260925 19:08:52.627493 20483 ts_manager.cc:194] Registered new tserver with Master: be33d00724774dc8879a3bd3db1a166e (127.0.196.195:44163)
I20260925 19:08:52.627921 20483 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.196.195:53565
I20260925 19:08:52.634240 787 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:08:52.639298 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:52.635856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53654 (local address 127.0.196.254:38399)
0925 19:08:52.635956 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:52.635958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:52.636063 (+ 105us) server_negotiation.cc:407] Connection header received
0925 19:08:52.636178 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:52.636179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:52.636193 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:52.636273 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:52.636452 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.636979 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.637806 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.638007 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.638090 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:52.638973 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:52.638974 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:52.638975 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:08:52.638981 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:52.639000 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:08:52.639004 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:52.639065 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:52.639166 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 19:08:52.639196 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 19:08:52.641518 20483 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:08:52.643625 20483 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:08:52.651916 20942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:52.646542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.195:44163 (local address 127.0.0.1:47374)
0925 19:08:52.646773 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:52.646785 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:52.646836 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:52.647678 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:52.647679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:52.647681 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:52.647723 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:52.647724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.649154 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.649156 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:52.650568 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:52.650571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.651435 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.651436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:52.651628 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:52.651652 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 19:08:52.651726 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 19:08:52.651786 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":8,"thread_start_us":97,"threads_started":1}
I20260925 19:08:52.652074 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:52.646521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53392 (local address 127.0.196.194:34471)
0925 19:08:52.646913 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:52.646918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:52.646931 (+ 13us) server_negotiation.cc:407] Connection header received
0925 19:08:52.647170 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:52.647174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:52.647192 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:52.647268 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:52.647495 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.648796 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.650701 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.651547 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.651618 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:52.651714 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:52.651909 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 19:08:52.651962 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":159,"threads_started":1}
I20260925 19:08:52.652307 20945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:52.646738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47374 (local address 127.0.196.195:44163)
0925 19:08:52.647402 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:52.647407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:52.647418 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:08:52.647459 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:52.647461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:52.647478 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:52.647578 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:52.647827 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.648994 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.650730 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.651332 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.651413 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:52.651516 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:52.652142 (+ 626us) server_negotiation.cc:299] Negotiation successful
0925 19:08:52.652212 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":115,"threads_started":1}
I20260925 19:08:52.652344 20941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:52.646312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:34471 (local address 127.0.0.1:53392)
0925 19:08:52.646477 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:52.646574 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:52.646687 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:52.647334 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:52.647335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:52.647336 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:52.647412 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:52.647416 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.648934 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.648935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:52.650568 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:52.650571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.651752 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.651753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:52.651810 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:52.651813 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:08:52.652201 (+ 388us) client_negotiation.cc:240] Negotiation successful
0925 19:08:52.652252 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":89,"thread_start_us":58,"threads_started":1}
I20260925 19:08:52.652814 20946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:52.646870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50272 (local address 127.0.196.193:33485)
0925 19:08:52.647061 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:52.647067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:52.647078 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:08:52.647125 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:52.647127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:52.647145 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:52.647238 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:52.648925 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.649952 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.651269 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.652102 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.652189 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:52.652276 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:52.652660 (+ 384us) server_negotiation.cc:299] Negotiation successful
0925 19:08:52.652715 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":78,"threads_started":1}
I20260925 19:08:52.653131 20867 tablet_service.cc:1467] Processing CreateTablet for tablet fabac3c6cf2142aba953c3241a061f8c (DEFAULT_TABLE table=test-table [id=ba9f4e2b04514b0cafaaa9d5fd5981fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:08:52.653282 20736 tablet_service.cc:1467] Processing CreateTablet for tablet fabac3c6cf2142aba953c3241a061f8c (DEFAULT_TABLE table=test-table [id=ba9f4e2b04514b0cafaaa9d5fd5981fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:08:52.653573 20867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fabac3c6cf2142aba953c3241a061f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:52.653606 20736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fabac3c6cf2142aba953c3241a061f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:52.652580 20944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:52.646744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:33485 (local address 127.0.0.1:50272)
0925 19:08:52.646993 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:52.647003 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:52.647044 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:52.647310 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:52.647313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:52.647316 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:52.647412 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:52.647414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.650085 (+ 2671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.650086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:52.651151 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:52.651154 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.652315 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.652316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:52.652382 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:52.652386 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:08:52.652443 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 19:08:52.652501 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":23,"thread_start_us":70,"threads_started":1}
I20260925 19:08:52.657771 20947 tablet_bootstrap.cc:492] T fabac3c6cf2142aba953c3241a061f8c P 03ca2a4171b64dc8ac94e62b3417c6d0: Bootstrap starting.
I20260925 19:08:52.658658 20947 tablet_bootstrap.cc:654] T fabac3c6cf2142aba953c3241a061f8c P 03ca2a4171b64dc8ac94e62b3417c6d0: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:52.658746 20605 tablet_service.cc:1467] Processing CreateTablet for tablet fabac3c6cf2142aba953c3241a061f8c (DEFAULT_TABLE table=test-table [id=ba9f4e2b04514b0cafaaa9d5fd5981fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:08:52.659034 20947 log.cc:824] T fabac3c6cf2142aba953c3241a061f8c P 03ca2a4171b64dc8ac94e62b3417c6d0: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:52.659180 20605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fabac3c6cf2142aba953c3241a061f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:52.659471 20948 tablet_bootstrap.cc:492] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e: Bootstrap starting.
I20260925 19:08:52.659914 20947 tablet_bootstrap.cc:492] T fabac3c6cf2142aba953c3241a061f8c P 03ca2a4171b64dc8ac94e62b3417c6d0: No bootstrap required, opened a new log
I20260925 19:08:52.659986 20947 ts_tablet_manager.cc:1397] T fabac3c6cf2142aba953c3241a061f8c P 03ca2a4171b64dc8ac94e62b3417c6d0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:08:52.660182 20948 tablet_bootstrap.cc:654] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:52.660517 20948 log.cc:824] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:52.661422 20948 tablet_bootstrap.cc:492] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e: No bootstrap required, opened a new log
I20260925 19:08:52.661499 20948 ts_tablet_manager.cc:1397] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:08:52.662370 20951 tablet_bootstrap.cc:492] T fabac3c6cf2142aba953c3241a061f8c P b275bde3f075438a9571bb5cb48211c4: Bootstrap starting.
I20260925 19:08:52.663033 20951 tablet_bootstrap.cc:654] T fabac3c6cf2142aba953c3241a061f8c P b275bde3f075438a9571bb5cb48211c4: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:52.663344 20951 log.cc:824] T fabac3c6cf2142aba953c3241a061f8c P b275bde3f075438a9571bb5cb48211c4: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:52.663308 20947 raft_consensus.cc:348] T fabac3c6cf2142aba953c3241a061f8c P 03ca2a4171b64dc8ac94e62b3417c6d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ca2a4171b64dc8ac94e62b3417c6d0" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34471 } } peers { permanent_uuid: "be33d00724774dc8879a3bd3db1a166e" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44163 } } peers { permanent_uuid: "b275bde3f075438a9571bb5cb48211c4" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 33485 } }
I20260925 19:08:52.663501 20947 raft_consensus.cc:374] T fabac3c6cf2142aba953c3241a061f8c P 03ca2a4171b64dc8ac94e62b3417c6d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:52.663540 20947 raft_consensus.cc:729] T fabac3c6cf2142aba953c3241a061f8c P 03ca2a4171b64dc8ac94e62b3417c6d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ca2a4171b64dc8ac94e62b3417c6d0, State: Initialized, Role: FOLLOWER
I20260925 19:08:52.663676 20947 consensus_queue.cc:260] T fabac3c6cf2142aba953c3241a061f8c P 03ca2a4171b64dc8ac94e62b3417c6d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ca2a4171b64dc8ac94e62b3417c6d0" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34471 } } peers { permanent_uuid: "be33d00724774dc8879a3bd3db1a166e" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44163 } } peers { permanent_uuid: "b275bde3f075438a9571bb5cb48211c4" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 33485 } }
I20260925 19:08:52.663965 20947 ts_tablet_manager.cc:1428] T fabac3c6cf2142aba953c3241a061f8c P 03ca2a4171b64dc8ac94e62b3417c6d0: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260925 19:08:52.664001 20802 heartbeater.cc:503] Master 127.0.196.254:38399 was elected leader, sending a full tablet report...
I20260925 19:08:52.664278 20948 raft_consensus.cc:348] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ca2a4171b64dc8ac94e62b3417c6d0" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34471 } } peers { permanent_uuid: "be33d00724774dc8879a3bd3db1a166e" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44163 } } peers { permanent_uuid: "b275bde3f075438a9571bb5cb48211c4" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 33485 } }
I20260925 19:08:52.664464 20948 raft_consensus.cc:374] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:52.664510 20948 raft_consensus.cc:729] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be33d00724774dc8879a3bd3db1a166e, State: Initialized, Role: FOLLOWER
I20260925 19:08:52.664728 20948 consensus_queue.cc:260] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ca2a4171b64dc8ac94e62b3417c6d0" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34471 } } peers { permanent_uuid: "be33d00724774dc8879a3bd3db1a166e" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44163 } } peers { permanent_uuid: "b275bde3f075438a9571bb5cb48211c4" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 33485 } }
I20260925 19:08:52.664845 20954 raft_consensus.cc:482] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:08:52.664932 20954 raft_consensus.cc:504] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ca2a4171b64dc8ac94e62b3417c6d0" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34471 } } peers { permanent_uuid: "be33d00724774dc8879a3bd3db1a166e" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44163 } } peers { permanent_uuid: "b275bde3f075438a9571bb5cb48211c4" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 33485 } }
I20260925 19:08:52.665205 20948 ts_tablet_manager.cc:1428] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260925 19:08:52.665328 20933 heartbeater.cc:503] Master 127.0.196.254:38399 was elected leader, sending a full tablet report...
I20260925 19:08:52.665560 20951 tablet_bootstrap.cc:492] T fabac3c6cf2142aba953c3241a061f8c P b275bde3f075438a9571bb5cb48211c4: No bootstrap required, opened a new log
I20260925 19:08:52.665635 20951 ts_tablet_manager.cc:1397] T fabac3c6cf2142aba953c3241a061f8c P b275bde3f075438a9571bb5cb48211c4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 19:08:52.666245 20954 leader_election.cc:288] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03ca2a4171b64dc8ac94e62b3417c6d0 (127.0.196.194:34471), b275bde3f075438a9571bb5cb48211c4 (127.0.196.193:33485)
I20260925 19:08:52.668143 20951 raft_consensus.cc:348] T fabac3c6cf2142aba953c3241a061f8c P b275bde3f075438a9571bb5cb48211c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ca2a4171b64dc8ac94e62b3417c6d0" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34471 } } peers { permanent_uuid: "be33d00724774dc8879a3bd3db1a166e" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44163 } } peers { permanent_uuid: "b275bde3f075438a9571bb5cb48211c4" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 33485 } }
I20260925 19:08:52.668345 20951 raft_consensus.cc:374] T fabac3c6cf2142aba953c3241a061f8c P b275bde3f075438a9571bb5cb48211c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:52.668387 20951 raft_consensus.cc:729] T fabac3c6cf2142aba953c3241a061f8c P b275bde3f075438a9571bb5cb48211c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b275bde3f075438a9571bb5cb48211c4, State: Initialized, Role: FOLLOWER
I20260925 19:08:52.668576 20951 consensus_queue.cc:260] T fabac3c6cf2142aba953c3241a061f8c P b275bde3f075438a9571bb5cb48211c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ca2a4171b64dc8ac94e62b3417c6d0" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34471 } } peers { permanent_uuid: "be33d00724774dc8879a3bd3db1a166e" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44163 } } peers { permanent_uuid: "b275bde3f075438a9571bb5cb48211c4" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 33485 } }
I20260925 19:08:52.668907 20671 heartbeater.cc:503] Master 127.0.196.254:38399 was elected leader, sending a full tablet report...
I20260925 19:08:52.669250 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:52.665405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:34471 (local address 127.0.196.195:41519)
0925 19:08:52.665493 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:52.665505 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:52.665549 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:52.665832 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:52.665833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:52.665835 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:52.665906 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:52.665907 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.666951 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.666952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:52.668154 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:52.668156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.668971 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.668972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:52.669046 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:52.669050 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:08:52.669099 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 19:08:52.669147 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":20}
I20260925 19:08:52.669479 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:52.665625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:41519 (local address 127.0.196.194:34471)
0925 19:08:52.665703 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:52.665705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:52.665712 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:08:52.665731 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:52.665732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:52.665742 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:52.665803 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:52.666001 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.666847 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.668253 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.668833 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.668907 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:52.668917 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:52.669344 (+ 427us) server_negotiation.cc:299] Negotiation successful
0925 19:08:52.669410 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 19:08:52.669353 20951 ts_tablet_manager.cc:1428] T fabac3c6cf2142aba953c3241a061f8c P b275bde3f075438a9571bb5cb48211c4: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260925 19:08:52.669819 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabac3c6cf2142aba953c3241a061f8c" candidate_uuid: "be33d00724774dc8879a3bd3db1a166e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ca2a4171b64dc8ac94e62b3417c6d0" is_pre_election: true
I20260925 19:08:52.669975 20756 raft_consensus.cc:2454] T fabac3c6cf2142aba953c3241a061f8c P 03ca2a4171b64dc8ac94e62b3417c6d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be33d00724774dc8879a3bd3db1a166e in term 0.
I20260925 19:08:52.670013 20946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:52.666186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.196.195:35341 (local address 127.0.196.193:33485)
0925 19:08:52.666225 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:52.666227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:52.666441 (+ 214us) server_negotiation.cc:407] Connection header received
0925 19:08:52.666480 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:52.666481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:52.666490 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:52.666537 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:08:52.666718 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.667407 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.668748 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.669392 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.669466 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:52.669496 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:52.669915 (+ 419us) server_negotiation.cc:299] Negotiation successful
0925 19:08:52.669957 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:08:52.670028 20956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:52.666016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:33485 (local address 127.0.196.195:35341)
0925 19:08:52.666354 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:52.666364 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:08:52.666418 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:08:52.666537 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:08:52.666538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:08:52.666540 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:08:52.666611 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:52.666612 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.667507 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.667507 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:52.668630 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:08:52.668632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.669745 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.669746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:08:52.669814 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:52.669818 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:08:52.669906 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 19:08:52.669948 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":76,"threads_started":1}
I20260925 19:08:52.670231 20821 leader_election.cc:302] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ca2a4171b64dc8ac94e62b3417c6d0, be33d00724774dc8879a3bd3db1a166e; no voters:
I20260925 19:08:52.670307 20625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabac3c6cf2142aba953c3241a061f8c" candidate_uuid: "be33d00724774dc8879a3bd3db1a166e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b275bde3f075438a9571bb5cb48211c4" is_pre_election: true
I20260925 19:08:52.670351 20955 raft_consensus.cc:2781] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:08:52.670413 20955 raft_consensus.cc:482] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:08:52.670429 20625 raft_consensus.cc:2454] T fabac3c6cf2142aba953c3241a061f8c P b275bde3f075438a9571bb5cb48211c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be33d00724774dc8879a3bd3db1a166e in term 0.
I20260925 19:08:52.670449 20955 raft_consensus.cc:3037] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:52.671228 20955 raft_consensus.cc:504] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ca2a4171b64dc8ac94e62b3417c6d0" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34471 } } peers { permanent_uuid: "be33d00724774dc8879a3bd3db1a166e" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44163 } } peers { permanent_uuid: "b275bde3f075438a9571bb5cb48211c4" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 33485 } }
I20260925 19:08:52.671378 20955 leader_election.cc:288] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e [CANDIDATE]: Term 1 election: Requested vote from peers 03ca2a4171b64dc8ac94e62b3417c6d0 (127.0.196.194:34471), b275bde3f075438a9571bb5cb48211c4 (127.0.196.193:33485)
I20260925 19:08:52.671550 20756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabac3c6cf2142aba953c3241a061f8c" candidate_uuid: "be33d00724774dc8879a3bd3db1a166e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ca2a4171b64dc8ac94e62b3417c6d0"
I20260925 19:08:52.671589 20625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabac3c6cf2142aba953c3241a061f8c" candidate_uuid: "be33d00724774dc8879a3bd3db1a166e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b275bde3f075438a9571bb5cb48211c4"
I20260925 19:08:52.671659 20756 raft_consensus.cc:3037] T fabac3c6cf2142aba953c3241a061f8c P 03ca2a4171b64dc8ac94e62b3417c6d0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:52.671702 20625 raft_consensus.cc:3037] T fabac3c6cf2142aba953c3241a061f8c P b275bde3f075438a9571bb5cb48211c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:52.672417 20756 raft_consensus.cc:2454] T fabac3c6cf2142aba953c3241a061f8c P 03ca2a4171b64dc8ac94e62b3417c6d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be33d00724774dc8879a3bd3db1a166e in term 1.
I20260925 19:08:52.672418 20625 raft_consensus.cc:2454] T fabac3c6cf2142aba953c3241a061f8c P b275bde3f075438a9571bb5cb48211c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be33d00724774dc8879a3bd3db1a166e in term 1.
I20260925 19:08:52.672608 20821 leader_election.cc:302] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ca2a4171b64dc8ac94e62b3417c6d0, be33d00724774dc8879a3bd3db1a166e; no voters:
I20260925 19:08:52.672736 20955 raft_consensus.cc:2781] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:08:52.672817 20955 raft_consensus.cc:686] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e [term 1 LEADER]: Becoming Leader. State: Replica: be33d00724774dc8879a3bd3db1a166e, State: Running, Role: LEADER
I20260925 19:08:52.672912 20955 consensus_queue.cc:237] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ca2a4171b64dc8ac94e62b3417c6d0" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34471 } } peers { permanent_uuid: "be33d00724774dc8879a3bd3db1a166e" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44163 } } peers { permanent_uuid: "b275bde3f075438a9571bb5cb48211c4" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 33485 } }
I20260925 19:08:52.673923 20483 catalog_manager.cc:5495] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e reported cstate change: term changed from 0 to 1, leader changed from <none> to be33d00724774dc8879a3bd3db1a166e (127.0.196.195). New cstate: current_term: 1 leader_uuid: "be33d00724774dc8879a3bd3db1a166e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ca2a4171b64dc8ac94e62b3417c6d0" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be33d00724774dc8879a3bd3db1a166e" member_type: VOTER last_known_addr { host: "127.0.196.195" port: 44163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b275bde3f075438a9571bb5cb48211c4" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 33485 } health_report { overall_health: UNKNOWN } } }
I20260925 19:08:52.687250 20945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:52.684742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47386 (local address 127.0.196.195:44163)
0925 19:08:52.684818 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:52.684820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:52.684828 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:08:52.684867 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:52.684868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:52.684887 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:52.684961 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:08:52.685147 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.685634 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.686471 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:52.686753 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:52.686810 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:52.686966 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:08:52.687069 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:08:52.687134 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:52.687167 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 19:08:52.687182 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 19:08:52.689267 20756 raft_consensus.cc:1261] T fabac3c6cf2142aba953c3241a061f8c P 03ca2a4171b64dc8ac94e62b3417c6d0 [term 1 FOLLOWER]: Refusing update from remote peer be33d00724774dc8879a3bd3db1a166e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:08:52.689267 20625 raft_consensus.cc:1261] T fabac3c6cf2142aba953c3241a061f8c P b275bde3f075438a9571bb5cb48211c4 [term 1 FOLLOWER]: Refusing update from remote peer be33d00724774dc8879a3bd3db1a166e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:08:52.689644 20954 consensus_queue.cc:1035] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b275bde3f075438a9571bb5cb48211c4" member_type: VOTER last_known_addr { host: "127.0.196.193" port: 33485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:08:52.689816 20955 consensus_queue.cc:1035] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ca2a4171b64dc8ac94e62b3417c6d0" member_type: VOTER last_known_addr { host: "127.0.196.194" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:08:52.692413 20960 mvcc.cc:205] Tried to move back new op lower bound from 7333328210692108288 to 7333328210629394432. Current Snapshot: MvccSnapshot[applied={T|T < 7333328210692108288}]
I20260925 19:08:52.692548 20961 mvcc.cc:205] Tried to move back new op lower bound from 7333328210692108288 to 7333328210629394432. Current Snapshot: MvccSnapshot[applied={T|T < 7333328210692108288}]
I20260925 19:08:52.693059 20962 mvcc.cc:205] Tried to move back new op lower bound from 7333328210692108288 to 7333328210629394432. Current Snapshot: MvccSnapshot[applied={T|T < 7333328210692108288}]
I20260925 19:08:52.694802 20483 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:08:52.694909 20483 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:08:52.695729 20483 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47b42efdbb194ce891495863b92d1417: Sending DeleteTablet for 3 replicas of tablet fabac3c6cf2142aba953c3241a061f8c
I20260925 19:08:52.696081 20736 tablet_service.cc:1514] Processing DeleteTablet for tablet fabac3c6cf2142aba953c3241a061f8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:08:52 UTC) from {username='slave'} at 127.0.0.1:53392
I20260925 19:08:52.696089 20867 tablet_service.cc:1514] Processing DeleteTablet for tablet fabac3c6cf2142aba953c3241a061f8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:08:52 UTC) from {username='slave'} at 127.0.0.1:47374
I20260925 19:08:52.696117 20605 tablet_service.cc:1514] Processing DeleteTablet for tablet fabac3c6cf2142aba953c3241a061f8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:08:52 UTC) from {username='slave'} at 127.0.0.1:50272
I20260925 19:08:52.696439 20970 tablet_replica.cc:330] T fabac3c6cf2142aba953c3241a061f8c P 03ca2a4171b64dc8ac94e62b3417c6d0: stopping tablet replica
I20260925 19:08:52.696441 20972 tablet_replica.cc:330] T fabac3c6cf2142aba953c3241a061f8c P b275bde3f075438a9571bb5cb48211c4: stopping tablet replica
I20260925 19:08:52.696537 20970 raft_consensus.cc:2229] T fabac3c6cf2142aba953c3241a061f8c P 03ca2a4171b64dc8ac94e62b3417c6d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:08:52.696542 20972 raft_consensus.cc:2229] T fabac3c6cf2142aba953c3241a061f8c P b275bde3f075438a9571bb5cb48211c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:08:52.696635 20970 raft_consensus.cc:2258] T fabac3c6cf2142aba953c3241a061f8c P 03ca2a4171b64dc8ac94e62b3417c6d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:08:52.696661 20972 raft_consensus.cc:2258] T fabac3c6cf2142aba953c3241a061f8c P b275bde3f075438a9571bb5cb48211c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:08:52.697067 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 20543
I20260925 19:08:52.697098 20970 ts_tablet_manager.cc:1905] T fabac3c6cf2142aba953c3241a061f8c P 03ca2a4171b64dc8ac94e62b3417c6d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:08:52.697108 20972 ts_tablet_manager.cc:1905] T fabac3c6cf2142aba953c3241a061f8c P b275bde3f075438a9571bb5cb48211c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:08:52.699062 20970 ts_tablet_manager.cc:1918] T fabac3c6cf2142aba953c3241a061f8c P 03ca2a4171b64dc8ac94e62b3417c6d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:08:52.699133 20970 log.cc:1196] T fabac3c6cf2142aba953c3241a061f8c P 03ca2a4171b64dc8ac94e62b3417c6d0: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-1/wal/wals/fabac3c6cf2142aba953c3241a061f8c
I20260925 19:08:52.699396 20970 ts_tablet_manager.cc:1939] T fabac3c6cf2142aba953c3241a061f8c P 03ca2a4171b64dc8ac94e62b3417c6d0: Deleting consensus metadata
I20260925 19:08:52.699582 20971 tablet_replica.cc:330] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e: stopping tablet replica
I20260925 19:08:52.699687 20971 raft_consensus.cc:2229] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:08:52.699823 20971 raft_consensus.cc:2258] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:08:52.700273 20971 ts_tablet_manager.cc:1905] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:08:52.701781 20971 ts_tablet_manager.cc:1918] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:08:52.701853 20971 log.cc:1196] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e: Deleting WAL directory at /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790363087086945-787-0/minicluster-data/ts-2/wal/wals/fabac3c6cf2142aba953c3241a061f8c
I20260925 19:08:52.702069 20971 ts_tablet_manager.cc:1939] T fabac3c6cf2142aba953c3241a061f8c P be33d00724774dc8879a3bd3db1a166e: Deleting consensus metadata
I20260925 19:08:52.702499 20468 catalog_manager.cc:4841] TS be33d00724774dc8879a3bd3db1a166e (127.0.196.195:44163): tablet fabac3c6cf2142aba953c3241a061f8c (table test-table [id=ba9f4e2b04514b0cafaaa9d5fd5981fb]) successfully deleted
W20260925 19:08:52.703437 20468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.196.193:33485 (error 108)
I20260925 19:08:52.703624 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 20674
I20260925 19:08:52.704689 20944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:52.703909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.193:33485 (local address 127.0.0.1:50278)
0925 19:08:52.704586 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:52.704640 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.193:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":649}
W20260925 19:08:52.704873 20468 catalog_manager.cc:4584] TS b275bde3f075438a9571bb5cb48211c4 (127.0.196.193:33485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fabac3c6cf2142aba953c3241a061f8c: Network error: Client connection negotiation failed: client connection to 127.0.196.193:33485: connect: Connection refused (error 111)
I20260925 19:08:52.708828 20944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:52.708617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.194:34471 (local address 127.0.0.1:53408)
0925 19:08:52.708761 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 19:08:52.708787 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.194:34471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260925 19:08:52.709225 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 20805
I20260925 19:08:52.714306 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 20452
2026-09-25T19:08:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (711 ms)
[----------] 6 tests from EncryptionPolicyTest (9191 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:08:52 dist-test-slave-rn98 krb5kdc[20976](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:08:52 dist-test-slave-rn98 krb5kdc[20976](info): set up 2 sockets
Sep 25 19:08:52 dist-test-slave-rn98 krb5kdc[20976](info): commencing operation
krb5kdc: starting...
W20260925 19:08:53.766845 787 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:08:53 dist-test-slave-rn98 krb5kdc[20976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:08:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:08:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:08:53.903779 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41429
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:08:53.990762 20989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:53.990998 20989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:53.991024 20989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:53.992229 20989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:08:53.992272 20989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:53.992286 20989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:53.992300 20989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:08:53.992316 20989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:08:53.993903 20989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41429
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:53.994123 20989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:53.994307 20989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:53.999802 20995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:53.999799 20997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:53.999828 20994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:54.000159 20989 server_base.cc:1034] running on GCE node
I20260925 19:08:54.000445 20989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:54.000768 20989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:54.001964 20989 hybrid_clock.cc:648] HybridClock initialized: now 1790363334001938 us; error 44 us; skew 500 ppm
Sep 25 19:08:54 dist-test-slave-rn98 krb5kdc[20976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:08:54.004869 20989 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:08:54.005352 20989 webserver.cc:460] Webserver started at http://127.0.0.1:44161/ using document root <none> and password file <none>
I20260925 19:08:54.005615 20989 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:54.005690 20989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:54.005847 20989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:54.007043 20989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "40c6adebb65f4c7eb05dd120eec10f06"
format_stamp: "Formatted at 2026-09-25 19:08:54 on dist-test-slave-rn98"
server_key: "cb6497d07f3fd4a0c551220750dc88f7"
server_key_iv: "f66157a3c7219ef846baf7df54872365"
server_key_version: "encryptionkey@0"
I20260925 19:08:54.007436 20989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "40c6adebb65f4c7eb05dd120eec10f06"
format_stamp: "Formatted at 2026-09-25 19:08:54 on dist-test-slave-rn98"
server_key: "cb6497d07f3fd4a0c551220750dc88f7"
server_key_iv: "f66157a3c7219ef846baf7df54872365"
server_key_version: "encryptionkey@0"
I20260925 19:08:54.008957 20989 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:08:54.009835 21004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:54.010056 20989 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:08:54.010172 20989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "40c6adebb65f4c7eb05dd120eec10f06"
format_stamp: "Formatted at 2026-09-25 19:08:54 on dist-test-slave-rn98"
server_key: "cb6497d07f3fd4a0c551220750dc88f7"
server_key_iv: "f66157a3c7219ef846baf7df54872365"
server_key_version: "encryptionkey@0"
I20260925 19:08:54.010269 20989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:54.033574 20989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:54.036273 20989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:54.036496 20989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:54.041445 20989 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41429
I20260925 19:08:54.041477 21056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41429 every 8 connection(s)
I20260925 19:08:54.041945 20989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:08:54.043046 21057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:54.045428 21057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40c6adebb65f4c7eb05dd120eec10f06: Bootstrap starting.
I20260925 19:08:54.046108 21057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40c6adebb65f4c7eb05dd120eec10f06: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:54.046350 21057 log.cc:824] T 00000000000000000000000000000000 P 40c6adebb65f4c7eb05dd120eec10f06: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:54.047364 21057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40c6adebb65f4c7eb05dd120eec10f06: No bootstrap required, opened a new log
I20260925 19:08:54.049219 21057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40c6adebb65f4c7eb05dd120eec10f06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c6adebb65f4c7eb05dd120eec10f06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41429 } }
I20260925 19:08:54.049388 21057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40c6adebb65f4c7eb05dd120eec10f06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:54.049415 21057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40c6adebb65f4c7eb05dd120eec10f06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c6adebb65f4c7eb05dd120eec10f06, State: Initialized, Role: FOLLOWER
I20260925 19:08:54.049692 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 20989
I20260925 19:08:54.049659 21057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40c6adebb65f4c7eb05dd120eec10f06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c6adebb65f4c7eb05dd120eec10f06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41429 } }
I20260925 19:08:54.049758 21057 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40c6adebb65f4c7eb05dd120eec10f06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:08:54.049803 21057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40c6adebb65f4c7eb05dd120eec10f06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:08:54.049824 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:08:54.049863 21057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40c6adebb65f4c7eb05dd120eec10f06 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:54.050061 787 external_mini_cluster.cc:1428] Setting key e14ebdfa5515fe8aef7b082d7af6a2dd
I20260925 19:08:54.050848 21057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40c6adebb65f4c7eb05dd120eec10f06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c6adebb65f4c7eb05dd120eec10f06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41429 } }
I20260925 19:08:54.050994 21057 leader_election.cc:302] T 00000000000000000000000000000000 P 40c6adebb65f4c7eb05dd120eec10f06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40c6adebb65f4c7eb05dd120eec10f06; no voters:
I20260925 19:08:54.051199 21057 leader_election.cc:288] T 00000000000000000000000000000000 P 40c6adebb65f4c7eb05dd120eec10f06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:08:54.051312 21062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40c6adebb65f4c7eb05dd120eec10f06 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:08:54.051506 21062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40c6adebb65f4c7eb05dd120eec10f06 [term 1 LEADER]: Becoming Leader. State: Replica: 40c6adebb65f4c7eb05dd120eec10f06, State: Running, Role: LEADER
I20260925 19:08:54.051512 21057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40c6adebb65f4c7eb05dd120eec10f06 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:08:54.051752 21062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40c6adebb65f4c7eb05dd120eec10f06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c6adebb65f4c7eb05dd120eec10f06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41429 } }
I20260925 19:08:54.052169 21063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40c6adebb65f4c7eb05dd120eec10f06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40c6adebb65f4c7eb05dd120eec10f06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c6adebb65f4c7eb05dd120eec10f06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41429 } } }
I20260925 19:08:54.052255 21063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40c6adebb65f4c7eb05dd120eec10f06 [sys.catalog]: This master's current role is: LEADER
I20260925 19:08:54.052434 21064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40c6adebb65f4c7eb05dd120eec10f06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40c6adebb65f4c7eb05dd120eec10f06. Latest consensus state: current_term: 1 leader_uuid: "40c6adebb65f4c7eb05dd120eec10f06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c6adebb65f4c7eb05dd120eec10f06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41429 } } }
I20260925 19:08:54.052551 21064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40c6adebb65f4c7eb05dd120eec10f06 [sys.catalog]: This master's current role is: LEADER
I20260925 19:08:54.052793 21066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:08:54.053661 21066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:08:54.056262 21066 catalog_manager.cc:1347] Generated new cluster ID: 6c406f6c19cb4980bd9d8a506c60e74e
I20260925 19:08:54.056329 21066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:08:54 dist-test-slave-rn98 krb5kdc[20976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 19:08:54.065088 21066 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:08:54.065693 21066 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:08:54.066937 21061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:54.050698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39765 (local address 127.0.0.1:41429)
0925 19:08:54.050943 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:54.050952 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:54.051012 (+ 60us) server_negotiation.cc:407] Connection header received
0925 19:08:54.051163 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:54.051170 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:54.051242 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:54.052268 (+ 1026us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:54.052399 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:54.054167 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:54.055340 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:54.055917 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:54.056062 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:54.061694 (+ 5632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:54.061700 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:54.061701 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:08:54.061718 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:54.064250 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:54.064862 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:54.064866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:54.064866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:54.064995 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:54.065311 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:54.065312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:54.065312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:54.065683 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:54.065893 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:54.066444 (+ 551us) server_negotiation.cc:299] Negotiation successful
0925 19:08:54.066593 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":127,"threads_started":1}
I20260925 19:08:54.068080 787 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:08:54.072352 21066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40c6adebb65f4c7eb05dd120eec10f06: Generated new TSK 0
I20260925 19:08:54.072584 21066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:08:54.078084 21061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:54.069764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47339 (local address 127.0.0.1:41429)
0925 19:08:54.069808 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:54.069811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:54.070062 (+ 251us) server_negotiation.cc:407] Connection header received
0925 19:08:54.070093 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:54.070094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:54.070106 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:54.070171 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:54.070919 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:54.071828 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:54.073038 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:54.073311 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:54.073376 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:54.075099 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:54.075101 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:54.075102 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:08:54.075108 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:54.076731 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:54.077226 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:54.077227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:54.077227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:54.077285 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:54.077544 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:54.077545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:54.077545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:54.077670 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:54.077751 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:54.077964 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 19:08:54.078014 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:08:54.080518 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 20989
W20260925 19:08:54.085240 20974 connection.cc:535] client connection to 127.0.0.1:41429 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41429): Connection reset by peer (error 104)
2026-09-25T19:08:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:08:54 dist-test-slave-rn98 krb5kdc[21088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:08:54 dist-test-slave-rn98 krb5kdc[21088](info): set up 2 sockets
Sep 25 19:08:54 dist-test-slave-rn98 krb5kdc[21088](info): commencing operation
krb5kdc: starting...
W20260925 19:08:55.123003 787 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:08:55 dist-test-slave-rn98 krb5kdc[21088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:08:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:08:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:08:55.254739 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38675
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:08:55.344859 21101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:55.345094 21101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:55.345120 21101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:55.346338 21101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:08:55.346385 21101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:55.346400 21101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:55.346412 21101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:08:55.346424 21101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:08:55.347996 21101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38675
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:55.348270 21101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:55.348529 21101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:55.354179 21106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:55.354190 21109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:55.354183 21101 server_base.cc:1034] running on GCE node
W20260925 19:08:55.354190 21107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:55.354815 21101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:55.355193 21101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:55.356419 21101 hybrid_clock.cc:648] HybridClock initialized: now 1790363335356413 us; error 40 us; skew 500 ppm
Sep 25 19:08:55 dist-test-slave-rn98 krb5kdc[21088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:08:55.362979 21101 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:08:55.363498 21101 webserver.cc:460] Webserver started at http://127.0.0.1:36879/ using document root <none> and password file <none>
I20260925 19:08:55.363768 21101 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:55.363842 21101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:55.363994 21101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:55.365103 21101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "50787af7342946caaa333a7d2912d941"
format_stamp: "Formatted at 2026-09-25 19:08:55 on dist-test-slave-rn98"
server_key: "9b98f1ce4bc3dd5080606a10b3fd29ec"
server_key_iv: "63a7f8b1356b4f79aac44762f8961283"
server_key_version: "encryptionkey@0"
I20260925 19:08:55.365474 21101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "50787af7342946caaa333a7d2912d941"
format_stamp: "Formatted at 2026-09-25 19:08:55 on dist-test-slave-rn98"
server_key: "9b98f1ce4bc3dd5080606a10b3fd29ec"
server_key_iv: "63a7f8b1356b4f79aac44762f8961283"
server_key_version: "encryptionkey@0"
I20260925 19:08:55.367007 21101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260925 19:08:55.367923 21116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:55.368208 21101 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:08:55.368323 21101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "50787af7342946caaa333a7d2912d941"
format_stamp: "Formatted at 2026-09-25 19:08:55 on dist-test-slave-rn98"
server_key: "9b98f1ce4bc3dd5080606a10b3fd29ec"
server_key_iv: "63a7f8b1356b4f79aac44762f8961283"
server_key_version: "encryptionkey@0"
I20260925 19:08:55.368408 21101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:55.404379 21101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:55.407138 21101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:55.407352 21101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:55.411790 21101 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38675
I20260925 19:08:55.411832 21168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38675 every 8 connection(s)
I20260925 19:08:55.412220 21101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:08:55.413136 21169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:55.415635 21169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50787af7342946caaa333a7d2912d941: Bootstrap starting.
I20260925 19:08:55.416208 21169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50787af7342946caaa333a7d2912d941: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:55.416447 21169 log.cc:824] T 00000000000000000000000000000000 P 50787af7342946caaa333a7d2912d941: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:55.417223 21169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50787af7342946caaa333a7d2912d941: No bootstrap required, opened a new log
I20260925 19:08:55.418970 21169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50787af7342946caaa333a7d2912d941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50787af7342946caaa333a7d2912d941" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38675 } }
I20260925 19:08:55.419145 21169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50787af7342946caaa333a7d2912d941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:55.419170 21169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50787af7342946caaa333a7d2912d941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50787af7342946caaa333a7d2912d941, State: Initialized, Role: FOLLOWER
I20260925 19:08:55.419337 21169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50787af7342946caaa333a7d2912d941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50787af7342946caaa333a7d2912d941" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38675 } }
I20260925 19:08:55.419422 21169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50787af7342946caaa333a7d2912d941 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:08:55.419447 21169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50787af7342946caaa333a7d2912d941 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:08:55.419476 21169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50787af7342946caaa333a7d2912d941 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:55.420045 21169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50787af7342946caaa333a7d2912d941 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50787af7342946caaa333a7d2912d941" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38675 } }
I20260925 19:08:55.420143 21169 leader_election.cc:302] T 00000000000000000000000000000000 P 50787af7342946caaa333a7d2912d941 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50787af7342946caaa333a7d2912d941; no voters:
I20260925 19:08:55.420282 21169 leader_election.cc:288] T 00000000000000000000000000000000 P 50787af7342946caaa333a7d2912d941 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:08:55.420377 21172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50787af7342946caaa333a7d2912d941 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:08:55.420586 21172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50787af7342946caaa333a7d2912d941 [term 1 LEADER]: Becoming Leader. State: Replica: 50787af7342946caaa333a7d2912d941, State: Running, Role: LEADER
I20260925 19:08:55.420611 21169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50787af7342946caaa333a7d2912d941 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:08:55.420706 21172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50787af7342946caaa333a7d2912d941 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50787af7342946caaa333a7d2912d941" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38675 } }
I20260925 19:08:55.421132 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 21101
I20260925 19:08:55.421156 21173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50787af7342946caaa333a7d2912d941 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50787af7342946caaa333a7d2912d941" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50787af7342946caaa333a7d2912d941" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38675 } } }
I20260925 19:08:55.421197 21174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50787af7342946caaa333a7d2912d941 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50787af7342946caaa333a7d2912d941. Latest consensus state: current_term: 1 leader_uuid: "50787af7342946caaa333a7d2912d941" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50787af7342946caaa333a7d2912d941" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38675 } } }
I20260925 19:08:55.421264 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:08:55.421257 21173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50787af7342946caaa333a7d2912d941 [sys.catalog]: This master's current role is: LEADER
I20260925 19:08:55.421262 21174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50787af7342946caaa333a7d2912d941 [sys.catalog]: This master's current role is: LEADER
I20260925 19:08:55.421468 787 external_mini_cluster.cc:1428] Setting key b1b2dbe461e9f77aaa4a403a99d703c6
I20260925 19:08:55.421470 21176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:08:55.422012 21176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:08:55.423980 21176 catalog_manager.cc:1347] Generated new cluster ID: a6becc28becd485aa205a24fe6b8f9f0
I20260925 19:08:55.424052 21176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:08:55.427690 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:55.422199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54365 (local address 127.0.0.1:38675)
0925 19:08:55.422494 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:55.422504 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:55.422560 (+ 56us) server_negotiation.cc:407] Connection header received
0925 19:08:55.422987 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:55.422997 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:55.423072 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:55.423220 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:55.423611 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:55.424712 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:55.425549 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:55.426039 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:55.426165 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:55.426995 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:55.426998 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:55.426999 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:08:55.427196 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:55.427227 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:08:55.427239 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:55.427343 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:55.427434 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 19:08:55.427467 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":71,"threads_started":1}
I20260925 19:08:55.428735 787 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:08:55.432564 21176 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:08:55.433279 21176 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:08:55.434059 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:55.430152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36275 (local address 127.0.0.1:38675)
0925 19:08:55.430195 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:55.430197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:55.430798 (+ 601us) server_negotiation.cc:407] Connection header received
0925 19:08:55.430877 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:55.430878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:55.430890 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:55.430948 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:55.431092 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:55.431596 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:55.432609 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:55.432839 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:55.432872 (+ 33us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:55.433811 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:55.433813 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:55.433813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:08:55.433819 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:55.433833 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:08:55.433837 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:55.433891 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:55.433950 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 19:08:55.433962 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:08:55.447288 21176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50787af7342946caaa333a7d2912d941: Generated new TSK 0
I20260925 19:08:55.447567 21176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:08:55.455463 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 21101
W20260925 19:08:55.460426 21086 connection.cc:535] client connection to 127.0.0.1:38675 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38675): Connection reset by peer (error 104)
2026-09-25T19:08:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1374 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:08:55 dist-test-slave-rn98 krb5kdc[21200](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:08:55 dist-test-slave-rn98 krb5kdc[21200](info): set up 2 sockets
Sep 25 19:08:55 dist-test-slave-rn98 krb5kdc[21200](info): commencing operation
krb5kdc: starting...
W20260925 19:08:56.542882 787 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:08:56 dist-test-slave-rn98 krb5kdc[21200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:08:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:08:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:08:56.675349 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41843
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:08:56.762277 21213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:56.762524 21213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:56.762552 21213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:56.763887 21213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:08:56.763943 21213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:56.763958 21213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:56.763971 21213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:08:56.763984 21213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:08:56.765857 21213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41843
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:56.766230 21213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:56.766520 21213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:08:56.772012 21213 server_base.cc:1034] running on GCE node
W20260925 19:08:56.771974 21219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:56.772001 21221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:56.771991 21218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:56.772411 21213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:56.772729 21213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:56.773883 21213 hybrid_clock.cc:648] HybridClock initialized: now 1790363336773856 us; error 46 us; skew 500 ppm
Sep 25 19:08:56 dist-test-slave-rn98 krb5kdc[21200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363336, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:08:56.776660 21213 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:08:56.777191 21213 webserver.cc:460] Webserver started at http://127.0.0.1:41637/ using document root <none> and password file <none>
I20260925 19:08:56.777441 21213 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:56.777523 21213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:56.777678 21213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:56.778879 21213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "884fd07eb649458581822c939dd2dc0b"
format_stamp: "Formatted at 2026-09-25 19:08:56 on dist-test-slave-rn98"
server_key: "fecd0c50612bde808329a546b7d4c515"
server_key_iv: "be4dbb1b9bcfe9fac01318a5eeb9618c"
server_key_version: "encryptionkey@0"
I20260925 19:08:56.779268 21213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "884fd07eb649458581822c939dd2dc0b"
format_stamp: "Formatted at 2026-09-25 19:08:56 on dist-test-slave-rn98"
server_key: "fecd0c50612bde808329a546b7d4c515"
server_key_iv: "be4dbb1b9bcfe9fac01318a5eeb9618c"
server_key_version: "encryptionkey@0"
I20260925 19:08:56.780686 21213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:08:56.781539 21228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:56.781736 21213 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:08:56.781847 21213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "884fd07eb649458581822c939dd2dc0b"
format_stamp: "Formatted at 2026-09-25 19:08:56 on dist-test-slave-rn98"
server_key: "fecd0c50612bde808329a546b7d4c515"
server_key_iv: "be4dbb1b9bcfe9fac01318a5eeb9618c"
server_key_version: "encryptionkey@0"
I20260925 19:08:56.781939 21213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:56.785632 21213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:56.788277 21213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:56.788466 21213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:56.793004 21213 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41843
I20260925 19:08:56.793089 21280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41843 every 8 connection(s)
I20260925 19:08:56.793489 21213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:08:56.794516 21281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:56.796933 21281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 884fd07eb649458581822c939dd2dc0b: Bootstrap starting.
I20260925 19:08:56.797484 21281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 884fd07eb649458581822c939dd2dc0b: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:56.797729 21281 log.cc:824] T 00000000000000000000000000000000 P 884fd07eb649458581822c939dd2dc0b: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:56.798410 21281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 884fd07eb649458581822c939dd2dc0b: No bootstrap required, opened a new log
I20260925 19:08:56.800045 21281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 884fd07eb649458581822c939dd2dc0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884fd07eb649458581822c939dd2dc0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41843 } }
I20260925 19:08:56.800215 21281 raft_consensus.cc:374] T 00000000000000000000000000000000 P 884fd07eb649458581822c939dd2dc0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:56.800238 21281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 884fd07eb649458581822c939dd2dc0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 884fd07eb649458581822c939dd2dc0b, State: Initialized, Role: FOLLOWER
I20260925 19:08:56.800349 21281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 884fd07eb649458581822c939dd2dc0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884fd07eb649458581822c939dd2dc0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41843 } }
I20260925 19:08:56.800407 21281 raft_consensus.cc:388] T 00000000000000000000000000000000 P 884fd07eb649458581822c939dd2dc0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:08:56.800429 21281 raft_consensus.cc:482] T 00000000000000000000000000000000 P 884fd07eb649458581822c939dd2dc0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:08:56.800513 21281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 884fd07eb649458581822c939dd2dc0b [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:56.800727 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 21213
I20260925 19:08:56.800853 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:08:56.801059 787 external_mini_cluster.cc:1428] Setting key d4e7267a4b01f4aaa9038f6c9dfeef3f
I20260925 19:08:56.801276 21281 raft_consensus.cc:504] T 00000000000000000000000000000000 P 884fd07eb649458581822c939dd2dc0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884fd07eb649458581822c939dd2dc0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41843 } }
I20260925 19:08:56.801409 21281 leader_election.cc:302] T 00000000000000000000000000000000 P 884fd07eb649458581822c939dd2dc0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 884fd07eb649458581822c939dd2dc0b; no voters:
I20260925 19:08:56.801602 21281 leader_election.cc:288] T 00000000000000000000000000000000 P 884fd07eb649458581822c939dd2dc0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:08:56.801846 21284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 884fd07eb649458581822c939dd2dc0b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:08:56.801898 21281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 884fd07eb649458581822c939dd2dc0b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:08:56.801983 21284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 884fd07eb649458581822c939dd2dc0b [term 1 LEADER]: Becoming Leader. State: Replica: 884fd07eb649458581822c939dd2dc0b, State: Running, Role: LEADER
I20260925 19:08:56.802084 21284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 884fd07eb649458581822c939dd2dc0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884fd07eb649458581822c939dd2dc0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41843 } }
I20260925 19:08:56.802920 21288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 884fd07eb649458581822c939dd2dc0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 884fd07eb649458581822c939dd2dc0b. Latest consensus state: current_term: 1 leader_uuid: "884fd07eb649458581822c939dd2dc0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884fd07eb649458581822c939dd2dc0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41843 } } }
I20260925 19:08:56.803037 21288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 884fd07eb649458581822c939dd2dc0b [sys.catalog]: This master's current role is: LEADER
I20260925 19:08:56.803318 21299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:08:56.803404 21286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 884fd07eb649458581822c939dd2dc0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "884fd07eb649458581822c939dd2dc0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884fd07eb649458581822c939dd2dc0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41843 } } }
I20260925 19:08:56.803678 21286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 884fd07eb649458581822c939dd2dc0b [sys.catalog]: This master's current role is: LEADER
I20260925 19:08:56.803970 21299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:08:56.804131 21287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:56.801630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51755 (local address 127.0.0.1:41843)
0925 19:08:56.801861 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:56.801871 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:56.802085 (+ 214us) server_negotiation.cc:407] Connection header received
0925 19:08:56.802337 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:56.802348 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:56.802381 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:56.802549 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:56.803550 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:56.803554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:56.803554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:08:56.803559 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:56.803609 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:08:56.803626 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:56.803693 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:56.803851 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 19:08:56.803906 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":79,"threads_started":1}
I20260925 19:08:56.805241 787 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:08:56.805977 21299 catalog_manager.cc:1347] Generated new cluster ID: f161b484062c426881a3c858ec4c2c36
I20260925 19:08:56.806044 21299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:08:56.814996 21299 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:08:56.815212 21299 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:08:56.816383 21287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:56.814915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39601 (local address 127.0.0.1:41843)
0925 19:08:56.814962 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:56.814964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:56.815186 (+ 222us) server_negotiation.cc:407] Connection header received
0925 19:08:56.815333 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:56.815334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:56.815340 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:56.815406 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:56.816125 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:56.816126 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:56.816126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:08:56.816130 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:56.816146 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:08:56.816151 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:56.816178 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:56.816289 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 19:08:56.816308 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:08:56.825610 21299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 884fd07eb649458581822c939dd2dc0b: Generated new TSK 0
I20260925 19:08:56.825899 21299 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:08:56.829746 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 21213
2026-09-25T19:08:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:08:56 dist-test-slave-rn98 krb5kdc[21312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:08:56 dist-test-slave-rn98 krb5kdc[21312](info): set up 2 sockets
Sep 25 19:08:56 dist-test-slave-rn98 krb5kdc[21312](info): commencing operation
krb5kdc: starting...
W20260925 19:08:57.890986 787 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:08:57 dist-test-slave-rn98 krb5kdc[21312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:08:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:08:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:08:58.038959 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44451
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:08:58.125170 21325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:58.125415 21325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:58.125442 21325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:58.126667 21325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:08:58.126708 21325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:58.126724 21325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:58.126736 21325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:08:58.126748 21325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:08:58.128244 21325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:58.128463 21325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:58.128644 21325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:58.133900 21330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:58.134065 21325 server_base.cc:1034] running on GCE node
W20260925 19:08:58.133895 21333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:58.133895 21331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:58.134452 21325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:58.134795 21325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:58.135994 21325 hybrid_clock.cc:648] HybridClock initialized: now 1790363338135977 us; error 53 us; skew 500 ppm
Sep 25 19:08:58 dist-test-slave-rn98 krb5kdc[21312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:08:58.138737 21325 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:08:58.139252 21325 webserver.cc:460] Webserver started at http://127.0.0.1:41027/ using document root <none> and password file <none>
I20260925 19:08:58.139452 21325 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:58.139494 21325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:58.139590 21325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:58.140758 21325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "49ef9fcd4d804b148c8f4612b0a3c8d0"
format_stamp: "Formatted at 2026-09-25 19:08:58 on dist-test-slave-rn98"
server_key: "174111597f8ba97e8d6c39a026871d3e"
server_key_iv: "e112996d255466ce64d86a4d6cab8e65"
server_key_version: "encryptionkey@0"
I20260925 19:08:58.141098 21325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "49ef9fcd4d804b148c8f4612b0a3c8d0"
format_stamp: "Formatted at 2026-09-25 19:08:58 on dist-test-slave-rn98"
server_key: "174111597f8ba97e8d6c39a026871d3e"
server_key_iv: "e112996d255466ce64d86a4d6cab8e65"
server_key_version: "encryptionkey@0"
I20260925 19:08:58.142462 21325 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:08:58.143347 21340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:58.143548 21325 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:08:58.143638 21325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "49ef9fcd4d804b148c8f4612b0a3c8d0"
format_stamp: "Formatted at 2026-09-25 19:08:58 on dist-test-slave-rn98"
server_key: "174111597f8ba97e8d6c39a026871d3e"
server_key_iv: "e112996d255466ce64d86a4d6cab8e65"
server_key_version: "encryptionkey@0"
I20260925 19:08:58.143703 21325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:58.153040 21325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:58.155651 21325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:58.155781 21325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:58.159832 21325 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44451
I20260925 19:08:58.160209 21392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44451 every 8 connection(s)
I20260925 19:08:58.160260 21325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:08:58.161029 21393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:58.163631 21393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49ef9fcd4d804b148c8f4612b0a3c8d0: Bootstrap starting.
I20260925 19:08:58.164196 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 21325
I20260925 19:08:58.164302 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:08:58.164427 21393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49ef9fcd4d804b148c8f4612b0a3c8d0: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:58.164527 787 external_mini_cluster.cc:1428] Setting key 3d6b3b7355a18354a746138a0cad3714
I20260925 19:08:58.164762 21393 log.cc:824] T 00000000000000000000000000000000 P 49ef9fcd4d804b148c8f4612b0a3c8d0: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:58.165680 21393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49ef9fcd4d804b148c8f4612b0a3c8d0: No bootstrap required, opened a new log
I20260925 19:08:58.168246 21393 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49ef9fcd4d804b148c8f4612b0a3c8d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ef9fcd4d804b148c8f4612b0a3c8d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44451 } }
I20260925 19:08:58.168429 21393 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49ef9fcd4d804b148c8f4612b0a3c8d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:58.168462 21393 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49ef9fcd4d804b148c8f4612b0a3c8d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ef9fcd4d804b148c8f4612b0a3c8d0, State: Initialized, Role: FOLLOWER
I20260925 19:08:58.168606 21393 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49ef9fcd4d804b148c8f4612b0a3c8d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ef9fcd4d804b148c8f4612b0a3c8d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44451 } }
I20260925 19:08:58.168679 21393 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49ef9fcd4d804b148c8f4612b0a3c8d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:08:58.168702 21393 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49ef9fcd4d804b148c8f4612b0a3c8d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:08:58.168768 21393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49ef9fcd4d804b148c8f4612b0a3c8d0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:58.169418 21393 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49ef9fcd4d804b148c8f4612b0a3c8d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ef9fcd4d804b148c8f4612b0a3c8d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44451 } }
I20260925 19:08:58.169533 21393 leader_election.cc:302] T 00000000000000000000000000000000 P 49ef9fcd4d804b148c8f4612b0a3c8d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49ef9fcd4d804b148c8f4612b0a3c8d0; no voters:
I20260925 19:08:58.169721 21393 leader_election.cc:288] T 00000000000000000000000000000000 P 49ef9fcd4d804b148c8f4612b0a3c8d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:08:58.169849 21398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49ef9fcd4d804b148c8f4612b0a3c8d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:08:58.169971 21393 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49ef9fcd4d804b148c8f4612b0a3c8d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:08:58.170055 21398 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49ef9fcd4d804b148c8f4612b0a3c8d0 [term 1 LEADER]: Becoming Leader. State: Replica: 49ef9fcd4d804b148c8f4612b0a3c8d0, State: Running, Role: LEADER
I20260925 19:08:58.170146 21398 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49ef9fcd4d804b148c8f4612b0a3c8d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ef9fcd4d804b148c8f4612b0a3c8d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44451 } }
I20260925 19:08:58.170598 21398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49ef9fcd4d804b148c8f4612b0a3c8d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49ef9fcd4d804b148c8f4612b0a3c8d0. Latest consensus state: current_term: 1 leader_uuid: "49ef9fcd4d804b148c8f4612b0a3c8d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ef9fcd4d804b148c8f4612b0a3c8d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44451 } } }
I20260925 19:08:58.170691 21399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49ef9fcd4d804b148c8f4612b0a3c8d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49ef9fcd4d804b148c8f4612b0a3c8d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49ef9fcd4d804b148c8f4612b0a3c8d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44451 } } }
I20260925 19:08:58.170708 21398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49ef9fcd4d804b148c8f4612b0a3c8d0 [sys.catalog]: This master's current role is: LEADER
I20260925 19:08:58.171025 21399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49ef9fcd4d804b148c8f4612b0a3c8d0 [sys.catalog]: This master's current role is: LEADER
I20260925 19:08:58.171190 21407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:08:58.172046 21407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 19:08:58 dist-test-slave-rn98 krb5kdc[21312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 19:08:58.174196 21407 catalog_manager.cc:1347] Generated new cluster ID: 0790cfccd2ad40ca924dd2f843d1a1cd
I20260925 19:08:58.174257 21407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:08:58.177217 21397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:58.165102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39555 (local address 127.0.0.1:44451)
0925 19:08:58.165316 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:58.165326 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:58.165397 (+ 71us) server_negotiation.cc:407] Connection header received
0925 19:08:58.166021 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:58.166030 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:58.166095 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:58.166255 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:58.166987 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:58.168474 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:58.169388 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:58.170063 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:58.173377 (+ 3314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:58.173528 (+ 151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:58.173532 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:58.173532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:08:58.173542 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:58.175502 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:58.175928 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:58.175931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:58.175931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:58.176041 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:58.176263 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:58.176264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:58.176264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:58.176530 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:58.176720 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:58.176830 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 19:08:58.176991 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
I20260925 19:08:58.178155 787 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:08:58.186190 21397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:58.179676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34747 (local address 127.0.0.1:44451)
0925 19:08:58.179715 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:58.179717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:58.179822 (+ 105us) server_negotiation.cc:407] Connection header received
0925 19:08:58.179897 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:58.179898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:58.179907 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:58.179962 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:58.180500 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:58.181009 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:58.181772 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:58.182015 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:58.182108 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:58.183444 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:58.183445 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:58.183446 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:08:58.183453 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:58.184971 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:58.185354 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:58.185356 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:58.185356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:58.185417 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:08:58.185646 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:08:58.185647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:08:58.185647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:08:58.185764 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:58.185871 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:58.185989 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 19:08:58.186079 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:08:58.187873 21407 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:08:58.188597 21407 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:08:58.203701 21407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49ef9fcd4d804b148c8f4612b0a3c8d0: Generated new TSK 0
I20260925 19:08:58.203985 21407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:08:58.207242 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 21325
2026-09-25T19:08:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:08:58 dist-test-slave-rn98 krb5kdc[21424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:08:58 dist-test-slave-rn98 krb5kdc[21424](info): set up 2 sockets
Sep 25 19:08:58 dist-test-slave-rn98 krb5kdc[21424](info): commencing operation
krb5kdc: starting...
W20260925 19:08:59.247184 787 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:08:59 dist-test-slave-rn98 krb5kdc[21424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:08:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:08:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:08:59.383455 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46291
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:08:59.472932 21437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:08:59.473167 21437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:08:59.473192 21437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:08:59.474472 21437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:08:59.474526 21437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:08:59.474540 21437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:08:59.474586 21437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:08:59.474641 21437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:08:59.476254 21437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:08:59.476557 21437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:08:59.476819 21437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:08:59.482064 21445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:59.482071 21442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:08:59.482076 21443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:08:59.482273 21437 server_base.cc:1034] running on GCE node
I20260925 19:08:59.482703 21437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:08:59.483029 21437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:08:59.484179 21437 hybrid_clock.cc:648] HybridClock initialized: now 1790363339484145 us; error 52 us; skew 500 ppm
Sep 25 19:08:59 dist-test-slave-rn98 krb5kdc[21424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:08:59.486992 21437 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:08:59.487561 21437 webserver.cc:460] Webserver started at http://127.0.0.1:40773/ using document root <none> and password file <none>
I20260925 19:08:59.487789 21437 fs_manager.cc:359] Metadata directory not provided
I20260925 19:08:59.487836 21437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:08:59.487938 21437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:08:59.489086 21437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "a1a3a494f1574091b1051c5cd204ba43"
format_stamp: "Formatted at 2026-09-25 19:08:59 on dist-test-slave-rn98"
server_key: "976b3f1d192eb2b11355577f6ee436c4"
server_key_iv: "40fd58e6eda32280b83abef1587a057d"
server_key_version: "encryptionkey@0"
I20260925 19:08:59.489434 21437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "a1a3a494f1574091b1051c5cd204ba43"
format_stamp: "Formatted at 2026-09-25 19:08:59 on dist-test-slave-rn98"
server_key: "976b3f1d192eb2b11355577f6ee436c4"
server_key_iv: "40fd58e6eda32280b83abef1587a057d"
server_key_version: "encryptionkey@0"
I20260925 19:08:59.491029 21437 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:08:59.491787 21452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:08:59.492024 21437 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:08:59.492108 21437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "a1a3a494f1574091b1051c5cd204ba43"
format_stamp: "Formatted at 2026-09-25 19:08:59 on dist-test-slave-rn98"
server_key: "976b3f1d192eb2b11355577f6ee436c4"
server_key_iv: "40fd58e6eda32280b83abef1587a057d"
server_key_version: "encryptionkey@0"
I20260925 19:08:59.492168 21437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:08:59.510491 21437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:08:59.512526 21437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:08:59.512645 21437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:08:59.516835 21437 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46291
I20260925 19:08:59.516870 21504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46291 every 8 connection(s)
I20260925 19:08:59.517311 21437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:08:59.518220 21505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:08:59.518694 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 21437
I20260925 19:08:59.518837 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:08:59.519039 787 external_mini_cluster.cc:1428] Setting key bd4115373304989b397f7d5544ce1cee
I20260925 19:08:59.521534 21505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1a3a494f1574091b1051c5cd204ba43: Bootstrap starting.
I20260925 19:08:59.522253 21505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1a3a494f1574091b1051c5cd204ba43: Neither blocks nor log segments found. Creating new log.
I20260925 19:08:59.522560 21505 log.cc:824] T 00000000000000000000000000000000 P a1a3a494f1574091b1051c5cd204ba43: Log is configured to *not* fsync() on all Append() calls
I20260925 19:08:59.523401 21505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1a3a494f1574091b1051c5cd204ba43: No bootstrap required, opened a new log
I20260925 19:08:59.525471 21508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:59.519608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36381 (local address 127.0.0.1:46291)
0925 19:08:59.519806 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:59.519817 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:59.519854 (+ 37us) server_negotiation.cc:407] Connection header received
0925 19:08:59.520420 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:59.520429 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:59.520491 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:59.520656 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:59.520888 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:59.522253 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:59.523180 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:59.523783 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:59.523907 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:59.524681 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:59.524686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:59.524687 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:08:59.524907 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:59.524960 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:08:59.524977 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:59.525044 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:59.525115 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 19:08:59.525229 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
I20260925 19:08:59.526120 21505 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1a3a494f1574091b1051c5cd204ba43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1a3a494f1574091b1051c5cd204ba43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46291 } }
I20260925 19:08:59.526311 21505 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1a3a494f1574091b1051c5cd204ba43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:08:59.526350 21505 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1a3a494f1574091b1051c5cd204ba43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1a3a494f1574091b1051c5cd204ba43, State: Initialized, Role: FOLLOWER
I20260925 19:08:59.526466 21505 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1a3a494f1574091b1051c5cd204ba43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1a3a494f1574091b1051c5cd204ba43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46291 } }
I20260925 19:08:59.526554 21505 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1a3a494f1574091b1051c5cd204ba43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:08:59.526598 21505 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1a3a494f1574091b1051c5cd204ba43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:08:59.526681 21505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1a3a494f1574091b1051c5cd204ba43 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:08:59.526824 787 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:08:59.527467 21505 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1a3a494f1574091b1051c5cd204ba43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1a3a494f1574091b1051c5cd204ba43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46291 } }
I20260925 19:08:59.527590 21505 leader_election.cc:302] T 00000000000000000000000000000000 P a1a3a494f1574091b1051c5cd204ba43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1a3a494f1574091b1051c5cd204ba43; no voters:
I20260925 19:08:59.527806 21505 leader_election.cc:288] T 00000000000000000000000000000000 P a1a3a494f1574091b1051c5cd204ba43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:08:59.527923 21510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1a3a494f1574091b1051c5cd204ba43 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:08:59.528082 21510 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1a3a494f1574091b1051c5cd204ba43 [term 1 LEADER]: Becoming Leader. State: Replica: a1a3a494f1574091b1051c5cd204ba43, State: Running, Role: LEADER
I20260925 19:08:59.528088 21505 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1a3a494f1574091b1051c5cd204ba43 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:08:59.528951 21510 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1a3a494f1574091b1051c5cd204ba43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1a3a494f1574091b1051c5cd204ba43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46291 } }
I20260925 19:08:59.529917 21511 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1a3a494f1574091b1051c5cd204ba43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1a3a494f1574091b1051c5cd204ba43. Latest consensus state: current_term: 1 leader_uuid: "a1a3a494f1574091b1051c5cd204ba43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1a3a494f1574091b1051c5cd204ba43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46291 } } }
I20260925 19:08:59.530061 21515 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1a3a494f1574091b1051c5cd204ba43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1a3a494f1574091b1051c5cd204ba43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1a3a494f1574091b1051c5cd204ba43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46291 } } }
I20260925 19:08:59.530437 21511 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1a3a494f1574091b1051c5cd204ba43 [sys.catalog]: This master's current role is: LEADER
I20260925 19:08:59.530448 21515 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1a3a494f1574091b1051c5cd204ba43 [sys.catalog]: This master's current role is: LEADER
W20260925 19:08:59.530813 21529 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a1a3a494f1574091b1051c5cd204ba43: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:08:59.530869 21529 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:08:59.530937 21531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:08:59.531127 21531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:08:59.532994 21508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:08:59.528492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56579 (local address 127.0.0.1:46291)
0925 19:08:59.528535 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:08:59.528538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:08:59.528786 (+ 248us) server_negotiation.cc:407] Connection header received
0925 19:08:59.528857 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:08:59.528859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:08:59.528872 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:08:59.528949 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:08:59.529315 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:59.530131 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:59.531325 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:08:59.531593 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:08:59.531677 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:08:59.532580 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:08:59.532582 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:08:59.532582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:08:59.532589 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:08:59.532604 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:08:59.532609 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:08:59.532666 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:08:59.532891 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 19:08:59.532917 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:08:59.533118 21531 catalog_manager.cc:1347] Generated new cluster ID: 13b2a2328286414b9849ceec57d66807
I20260925 19:08:59.533147 21531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:08:59.568188 21531 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:08:59.568902 21531 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:08:59.576402 21531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1a3a494f1574091b1051c5cd204ba43: Generated new TSK 0
I20260925 19:08:59.576659 21531 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:08:59.588384 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 21437
2026-09-25T19:08:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1381 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:08:59 dist-test-slave-rn98 krb5kdc[21536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:08:59 dist-test-slave-rn98 krb5kdc[21536](info): set up 2 sockets
Sep 25 19:08:59 dist-test-slave-rn98 krb5kdc[21536](info): commencing operation
krb5kdc: starting...
W20260925 19:09:00.626850 787 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:09:00 dist-test-slave-rn98 krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:09:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:09:00.755429 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41561
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:00.838086 21549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:00.838335 21549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:00.838366 21549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:00.839656 21549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:00.839710 21549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:00.839725 21549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:00.839737 21549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:00.839749 21549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:00.841370 21549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:00.841662 21549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:00.841969 21549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:00.847826 21554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:00.847889 21555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:00.848035 21557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:00.848050 21549 server_base.cc:1034] running on GCE node
I20260925 19:09:00.848369 21549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:00.848690 21549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:00.849891 21549 hybrid_clock.cc:648] HybridClock initialized: now 1790363340849876 us; error 57 us; skew 500 ppm
Sep 25 19:09:00 dist-test-slave-rn98 krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:00.856523 21549 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:09:00.857048 21549 webserver.cc:460] Webserver started at http://127.0.0.1:39137/ using document root <none> and password file <none>
I20260925 19:09:00.857321 21549 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:00.857399 21549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:00.857553 21549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:00.858810 21549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "0ee7d784d5694b378ef1ecded6202d09"
format_stamp: "Formatted at 2026-09-25 19:09:00 on dist-test-slave-rn98"
server_key: "f00149999b4b899864a89d62e2ed5e03"
server_key_iv: "8b9215233cc72a9d30f417d1bfb78cb1"
server_key_version: "encryptionkey@0"
I20260925 19:09:00.859193 21549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "0ee7d784d5694b378ef1ecded6202d09"
format_stamp: "Formatted at 2026-09-25 19:09:00 on dist-test-slave-rn98"
server_key: "f00149999b4b899864a89d62e2ed5e03"
server_key_iv: "8b9215233cc72a9d30f417d1bfb78cb1"
server_key_version: "encryptionkey@0"
I20260925 19:09:00.860641 21549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 19:09:00.861485 21564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:00.861713 21549 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:09:00.861840 21549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "0ee7d784d5694b378ef1ecded6202d09"
format_stamp: "Formatted at 2026-09-25 19:09:00 on dist-test-slave-rn98"
server_key: "f00149999b4b899864a89d62e2ed5e03"
server_key_iv: "8b9215233cc72a9d30f417d1bfb78cb1"
server_key_version: "encryptionkey@0"
I20260925 19:09:00.861923 21549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:00.865449 21549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:00.867909 21549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:00.868078 21549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:00.872217 21549 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41561
I20260925 19:09:00.872330 21616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41561 every 8 connection(s)
I20260925 19:09:00.872640 21549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:00.873688 21617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:00.876240 21617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ee7d784d5694b378ef1ecded6202d09: Bootstrap starting.
I20260925 19:09:00.876895 21617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ee7d784d5694b378ef1ecded6202d09: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:00.877218 21617 log.cc:824] T 00000000000000000000000000000000 P 0ee7d784d5694b378ef1ecded6202d09: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:00.878058 21617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ee7d784d5694b378ef1ecded6202d09: No bootstrap required, opened a new log
I20260925 19:09:00.879853 21617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ee7d784d5694b378ef1ecded6202d09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ee7d784d5694b378ef1ecded6202d09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41561 } }
I20260925 19:09:00.879997 21617 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ee7d784d5694b378ef1ecded6202d09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:00.880020 21617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ee7d784d5694b378ef1ecded6202d09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ee7d784d5694b378ef1ecded6202d09, State: Initialized, Role: FOLLOWER
I20260925 19:09:00.880195 21617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ee7d784d5694b378ef1ecded6202d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ee7d784d5694b378ef1ecded6202d09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41561 } }
I20260925 19:09:00.880261 21617 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ee7d784d5694b378ef1ecded6202d09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:00.880285 21617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ee7d784d5694b378ef1ecded6202d09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:00.880312 21617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ee7d784d5694b378ef1ecded6202d09 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:00.880695 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 21549
I20260925 19:09:00.880856 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:09:00.881044 787 external_mini_cluster.cc:1428] Setting key da2b63b3b161a3b24e82b748c8c77429
I20260925 19:09:00.881026 21617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ee7d784d5694b378ef1ecded6202d09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ee7d784d5694b378ef1ecded6202d09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41561 } }
I20260925 19:09:00.881150 21617 leader_election.cc:302] T 00000000000000000000000000000000 P 0ee7d784d5694b378ef1ecded6202d09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ee7d784d5694b378ef1ecded6202d09; no voters:
I20260925 19:09:00.881367 21617 leader_election.cc:288] T 00000000000000000000000000000000 P 0ee7d784d5694b378ef1ecded6202d09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:00.881393 21620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ee7d784d5694b378ef1ecded6202d09 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:00.881505 21620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ee7d784d5694b378ef1ecded6202d09 [term 1 LEADER]: Becoming Leader. State: Replica: 0ee7d784d5694b378ef1ecded6202d09, State: Running, Role: LEADER
I20260925 19:09:00.881639 21620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ee7d784d5694b378ef1ecded6202d09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ee7d784d5694b378ef1ecded6202d09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41561 } }
I20260925 19:09:00.881640 21617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ee7d784d5694b378ef1ecded6202d09 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:00.883347 21621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ee7d784d5694b378ef1ecded6202d09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ee7d784d5694b378ef1ecded6202d09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ee7d784d5694b378ef1ecded6202d09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41561 } } }
I20260925 19:09:00.883486 21621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ee7d784d5694b378ef1ecded6202d09 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:00.883687 21624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:00.881677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60581 (local address 127.0.0.1:41561)
0925 19:09:00.881854 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:00.881864 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:00.881897 (+ 33us) server_negotiation.cc:407] Connection header received
0925 19:09:00.882023 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:00.882030 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:00.882057 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:00.882195 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:00.882999 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:00.883003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:00.883004 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:00.883008 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:00.883055 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:00.883073 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:00.883116 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:00.883275 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 19:09:00.883330 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260925 19:09:00.883381 21622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ee7d784d5694b378ef1ecded6202d09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ee7d784d5694b378ef1ecded6202d09. Latest consensus state: current_term: 1 leader_uuid: "0ee7d784d5694b378ef1ecded6202d09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ee7d784d5694b378ef1ecded6202d09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41561 } } }
I20260925 19:09:00.883977 21622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ee7d784d5694b378ef1ecded6202d09 [sys.catalog]: This master's current role is: LEADER
W20260925 19:09:00.884032 21636 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ee7d784d5694b378ef1ecded6202d09: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:09:00.884099 21636 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:09:00.884261 21638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:00.884444 21638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:00.884941 787 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:09:00.886876 21638 catalog_manager.cc:1347] Generated new cluster ID: 5f3784872d8a4dc5b2a360469827d215
I20260925 19:09:00.886933 21638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:09:00.887930 21624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:00.886558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60455 (local address 127.0.0.1:41561)
0925 19:09:00.886600 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:00.886602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:00.886815 (+ 213us) server_negotiation.cc:407] Connection header received
0925 19:09:00.886903 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:00.886904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:00.886909 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:00.886972 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:00.887688 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:00.887689 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:00.887689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:00.887694 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:00.887711 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:00.887716 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:00.887780 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:00.887847 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 19:09:00.887861 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:09:00.892086 21638 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:00.892277 21638 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:00.900072 21638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ee7d784d5694b378ef1ecded6202d09: Generated new TSK 0
I20260925 19:09:00.900336 21638 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:09:00.907884 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 21549
2026-09-25T19:09:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:09:00 dist-test-slave-rn98 krb5kdc[21648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:09:00 dist-test-slave-rn98 krb5kdc[21648](info): set up 2 sockets
Sep 25 19:09:00 dist-test-slave-rn98 krb5kdc[21648](info): commencing operation
krb5kdc: starting...
W20260925 19:09:01.962932 787 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:09:02 dist-test-slave-rn98 krb5kdc[21648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:09:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:09:02.094002 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44901
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:02.180249 21661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:02.180491 21661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:02.180517 21661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:02.181806 21661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:02.181861 21661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:02.181876 21661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:02.181890 21661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:02.181903 21661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:02.183557 21661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:02.183856 21661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:02.184105 21661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:02.189460 21669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:02.189445 21667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:02.189445 21666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:02.189900 21661 server_base.cc:1034] running on GCE node
I20260925 19:09:02.190117 21661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:02.190428 21661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:02.191591 21661 hybrid_clock.cc:648] HybridClock initialized: now 1790363342191578 us; error 31 us; skew 500 ppm
Sep 25 19:09:02 dist-test-slave-rn98 krb5kdc[21648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:02.194759 21661 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:09:02.195322 21661 webserver.cc:460] Webserver started at http://127.0.0.1:42431/ using document root <none> and password file <none>
I20260925 19:09:02.195570 21661 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:02.195621 21661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:02.195780 21661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:02.196957 21661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "b9cac24198f2488bb98bc7d0b4b346b5"
format_stamp: "Formatted at 2026-09-25 19:09:02 on dist-test-slave-rn98"
server_key: "4456c9e954b712612a7821f100b6852c"
server_key_iv: "8a880fcbcf5617291f06f5ca4a9cfaf9"
server_key_version: "encryptionkey@0"
I20260925 19:09:02.197340 21661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "b9cac24198f2488bb98bc7d0b4b346b5"
format_stamp: "Formatted at 2026-09-25 19:09:02 on dist-test-slave-rn98"
server_key: "4456c9e954b712612a7821f100b6852c"
server_key_iv: "8a880fcbcf5617291f06f5ca4a9cfaf9"
server_key_version: "encryptionkey@0"
I20260925 19:09:02.198812 21661 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:09:02.199828 21676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:02.200053 21661 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:02.200203 21661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "b9cac24198f2488bb98bc7d0b4b346b5"
format_stamp: "Formatted at 2026-09-25 19:09:02 on dist-test-slave-rn98"
server_key: "4456c9e954b712612a7821f100b6852c"
server_key_iv: "8a880fcbcf5617291f06f5ca4a9cfaf9"
server_key_version: "encryptionkey@0"
I20260925 19:09:02.200280 21661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:02.215535 21661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:02.217891 21661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:02.218096 21661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:02.222890 21661 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44901
I20260925 19:09:02.222923 21728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44901 every 8 connection(s)
I20260925 19:09:02.223341 21661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:02.224227 21729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:02.226954 21729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9cac24198f2488bb98bc7d0b4b346b5: Bootstrap starting.
I20260925 19:09:02.227563 21729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9cac24198f2488bb98bc7d0b4b346b5: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:02.227860 21729 log.cc:824] T 00000000000000000000000000000000 P b9cac24198f2488bb98bc7d0b4b346b5: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:02.228641 21729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9cac24198f2488bb98bc7d0b4b346b5: No bootstrap required, opened a new log
I20260925 19:09:02.229718 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 21661
I20260925 19:09:02.229856 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:09:02.230072 787 external_mini_cluster.cc:1428] Setting key 6e7ce3c37e9d384b00520bdb2a9caf06
I20260925 19:09:02.230808 21729 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9cac24198f2488bb98bc7d0b4b346b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9cac24198f2488bb98bc7d0b4b346b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44901 } }
I20260925 19:09:02.230978 21729 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9cac24198f2488bb98bc7d0b4b346b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:02.231003 21729 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9cac24198f2488bb98bc7d0b4b346b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9cac24198f2488bb98bc7d0b4b346b5, State: Initialized, Role: FOLLOWER
I20260925 19:09:02.231148 21729 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9cac24198f2488bb98bc7d0b4b346b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9cac24198f2488bb98bc7d0b4b346b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44901 } }
I20260925 19:09:02.231221 21729 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9cac24198f2488bb98bc7d0b4b346b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:02.231243 21729 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9cac24198f2488bb98bc7d0b4b346b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:02.231364 21729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9cac24198f2488bb98bc7d0b4b346b5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:02.232017 21729 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9cac24198f2488bb98bc7d0b4b346b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9cac24198f2488bb98bc7d0b4b346b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44901 } }
I20260925 19:09:02.232187 21729 leader_election.cc:302] T 00000000000000000000000000000000 P b9cac24198f2488bb98bc7d0b4b346b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9cac24198f2488bb98bc7d0b4b346b5; no voters:
I20260925 19:09:02.232375 21729 leader_election.cc:288] T 00000000000000000000000000000000 P b9cac24198f2488bb98bc7d0b4b346b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:02.232412 21734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9cac24198f2488bb98bc7d0b4b346b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:02.232661 21734 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9cac24198f2488bb98bc7d0b4b346b5 [term 1 LEADER]: Becoming Leader. State: Replica: b9cac24198f2488bb98bc7d0b4b346b5, State: Running, Role: LEADER
I20260925 19:09:02.232705 21729 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9cac24198f2488bb98bc7d0b4b346b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:02.232836 21734 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9cac24198f2488bb98bc7d0b4b346b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9cac24198f2488bb98bc7d0b4b346b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44901 } }
I20260925 19:09:02.233233 21736 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9cac24198f2488bb98bc7d0b4b346b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9cac24198f2488bb98bc7d0b4b346b5. Latest consensus state: current_term: 1 leader_uuid: "b9cac24198f2488bb98bc7d0b4b346b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9cac24198f2488bb98bc7d0b4b346b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44901 } } }
I20260925 19:09:02.233641 21736 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9cac24198f2488bb98bc7d0b4b346b5 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:02.233544 21735 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9cac24198f2488bb98bc7d0b4b346b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9cac24198f2488bb98bc7d0b4b346b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9cac24198f2488bb98bc7d0b4b346b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44901 } } }
I20260925 19:09:02.233850 21735 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9cac24198f2488bb98bc7d0b4b346b5 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:02.233865 21743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:02.234489 21743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:02.236342 21743 catalog_manager.cc:1347] Generated new cluster ID: c7297aae21234fafa4d05dbb057aaaf9
I20260925 19:09:02.236395 21743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:09:02 dist-test-slave-rn98 krb5kdc[21648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 19:09:02.245874 21733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:02.230718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:45847 (local address 127.0.0.1:44901)
0925 19:09:02.230931 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:02.230939 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:02.230967 (+ 28us) server_negotiation.cc:407] Connection header received
0925 19:09:02.231113 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:02.231126 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:02.231252 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:02.231420 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:02.232322 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:02.234111 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:02.235196 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:02.235828 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:02.235958 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:02.241321 (+ 5363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:02.241328 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:02.241328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:02.241344 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:02.243601 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:02.244204 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:02.244209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:02.244209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:02.244462 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:02.244811 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:02.244812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:02.244813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:02.245168 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:02.245328 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:02.245460 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 19:09:02.245603 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":103,"threads_started":1}
I20260925 19:09:02.247046 787 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:09:02.248818 21743 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:02.249533 21743 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:02.253616 21743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9cac24198f2488bb98bc7d0b4b346b5: Generated new TSK 0
I20260925 19:09:02.253857 21743 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:09:02.256081 21733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:02.248583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:36897 (local address 127.0.0.1:44901)
0925 19:09:02.248623 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:02.248625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:02.248831 (+ 206us) server_negotiation.cc:407] Connection header received
0925 19:09:02.248909 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:02.248910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:02.248921 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:02.248976 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:02.249561 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:02.250054 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:02.250911 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:02.251130 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:02.251216 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:02.252670 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:02.252672 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:02.252672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:02.252681 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:02.254694 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:02.255182 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:02.255184 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:02.255184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:02.255256 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:02.255497 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:02.255498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:02.255498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:02.255639 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:02.255756 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:02.255914 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 19:09:02.255985 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:09:02.258401 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 21661
W20260925 19:09:02.262876 21646 connection.cc:535] client connection to 127.0.0.1:44901 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44901): Connection reset by peer (error 104)
2026-09-25T19:09:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1350 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:09:02 dist-test-slave-rn98 krb5kdc[21760](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:09:02 dist-test-slave-rn98 krb5kdc[21760](info): set up 2 sockets
Sep 25 19:09:02 dist-test-slave-rn98 krb5kdc[21760](info): commencing operation
krb5kdc: starting...
W20260925 19:09:03.326884 787 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:09:03 dist-test-slave-rn98 krb5kdc[21760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:09:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:09:03.454457 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35699
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:03.538182 21773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:03.538408 21773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:03.538434 21773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:03.539800 21773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:03.539856 21773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:03.539870 21773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:03.539883 21773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:03.539896 21773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:03.541555 21773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:03.541850 21773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:03.542109 21773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:03.547551 21779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:03.547614 21773 server_base.cc:1034] running on GCE node
W20260925 19:09:03.547566 21778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:03.547566 21781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:03.548063 21773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:03.548405 21773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:03.549578 21773 hybrid_clock.cc:648] HybridClock initialized: now 1790363343549577 us; error 40 us; skew 500 ppm
Sep 25 19:09:03 dist-test-slave-rn98 krb5kdc[21760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:03.552384 21773 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:09:03.552929 21773 webserver.cc:460] Webserver started at http://127.0.0.1:44989/ using document root <none> and password file <none>
I20260925 19:09:03.553195 21773 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:03.553278 21773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:03.553428 21773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:03.554548 21773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "7b7ad57fe7a243a29cf1f549d9f2c885"
format_stamp: "Formatted at 2026-09-25 19:09:03 on dist-test-slave-rn98"
server_key: "cd9d3fd89b5ba742ccb5ac84e0d81c11"
server_key_iv: "fd5430fd78c7c3c829e26da0657885f4"
server_key_version: "encryptionkey@0"
I20260925 19:09:03.555001 21773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "7b7ad57fe7a243a29cf1f549d9f2c885"
format_stamp: "Formatted at 2026-09-25 19:09:03 on dist-test-slave-rn98"
server_key: "cd9d3fd89b5ba742ccb5ac84e0d81c11"
server_key_iv: "fd5430fd78c7c3c829e26da0657885f4"
server_key_version: "encryptionkey@0"
I20260925 19:09:03.556444 21773 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:03.557297 21788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:03.557504 21773 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:03.557629 21773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "7b7ad57fe7a243a29cf1f549d9f2c885"
format_stamp: "Formatted at 2026-09-25 19:09:03 on dist-test-slave-rn98"
server_key: "cd9d3fd89b5ba742ccb5ac84e0d81c11"
server_key_iv: "fd5430fd78c7c3c829e26da0657885f4"
server_key_version: "encryptionkey@0"
I20260925 19:09:03.557719 21773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:03.591893 21773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:03.594478 21773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:03.594693 21773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:03.599504 21773 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35699
I20260925 19:09:03.599558 21840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35699 every 8 connection(s)
I20260925 19:09:03.600008 21773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:03.600915 21841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:03.603451 21841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b7ad57fe7a243a29cf1f549d9f2c885: Bootstrap starting.
I20260925 19:09:03.604108 21841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b7ad57fe7a243a29cf1f549d9f2c885: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:03.604435 21841 log.cc:824] T 00000000000000000000000000000000 P 7b7ad57fe7a243a29cf1f549d9f2c885: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:03.605288 21841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b7ad57fe7a243a29cf1f549d9f2c885: No bootstrap required, opened a new log
I20260925 19:09:03.606983 21841 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b7ad57fe7a243a29cf1f549d9f2c885 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7ad57fe7a243a29cf1f549d9f2c885" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35699 } }
I20260925 19:09:03.607138 21841 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b7ad57fe7a243a29cf1f549d9f2c885 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:03.607163 21841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b7ad57fe7a243a29cf1f549d9f2c885 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7ad57fe7a243a29cf1f549d9f2c885, State: Initialized, Role: FOLLOWER
I20260925 19:09:03.607352 21841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b7ad57fe7a243a29cf1f549d9f2c885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7ad57fe7a243a29cf1f549d9f2c885" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35699 } }
I20260925 19:09:03.607426 21841 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b7ad57fe7a243a29cf1f549d9f2c885 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:03.607450 21841 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b7ad57fe7a243a29cf1f549d9f2c885 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:03.607502 21841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b7ad57fe7a243a29cf1f549d9f2c885 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:03.608127 21841 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b7ad57fe7a243a29cf1f549d9f2c885 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7ad57fe7a243a29cf1f549d9f2c885" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35699 } }
I20260925 19:09:03.608220 21841 leader_election.cc:302] T 00000000000000000000000000000000 P 7b7ad57fe7a243a29cf1f549d9f2c885 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b7ad57fe7a243a29cf1f549d9f2c885; no voters:
I20260925 19:09:03.608433 21841 leader_election.cc:288] T 00000000000000000000000000000000 P 7b7ad57fe7a243a29cf1f549d9f2c885 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:03.608484 21844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b7ad57fe7a243a29cf1f549d9f2c885 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:03.608686 21844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b7ad57fe7a243a29cf1f549d9f2c885 [term 1 LEADER]: Becoming Leader. State: Replica: 7b7ad57fe7a243a29cf1f549d9f2c885, State: Running, Role: LEADER
I20260925 19:09:03.608738 21841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b7ad57fe7a243a29cf1f549d9f2c885 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:03.608812 21844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b7ad57fe7a243a29cf1f549d9f2c885 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7ad57fe7a243a29cf1f549d9f2c885" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35699 } }
I20260925 19:09:03.609246 21845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b7ad57fe7a243a29cf1f549d9f2c885 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b7ad57fe7a243a29cf1f549d9f2c885" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7ad57fe7a243a29cf1f549d9f2c885" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35699 } } }
I20260925 19:09:03.609287 21846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b7ad57fe7a243a29cf1f549d9f2c885 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b7ad57fe7a243a29cf1f549d9f2c885. Latest consensus state: current_term: 1 leader_uuid: "7b7ad57fe7a243a29cf1f549d9f2c885" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7ad57fe7a243a29cf1f549d9f2c885" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35699 } } }
I20260925 19:09:03.609421 21845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b7ad57fe7a243a29cf1f549d9f2c885 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:03.609440 21846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b7ad57fe7a243a29cf1f549d9f2c885 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:03.609835 21849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:03.610174 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 21773
I20260925 19:09:03.610319 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:09:03.610466 21849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:03.610539 787 external_mini_cluster.cc:1428] Setting key e7b715f2b1718d68e69f86aecaf2363b
I20260925 19:09:03.612571 21849 catalog_manager.cc:1347] Generated new cluster ID: 52da0c6cee1a4b7581570aee03c1a426
I20260925 19:09:03.612634 21849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:09:03.617276 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:03.611269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:40193 (local address 127.0.0.1:35699)
0925 19:09:03.611464 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:03.611474 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:03.611510 (+ 36us) server_negotiation.cc:407] Connection header received
0925 19:09:03.611636 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:03.611644 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:03.611704 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:03.611867 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:03.612125 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:03.613564 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:03.614683 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:03.615392 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:03.615517 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:03.616345 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:03.616351 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:03.616351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:03.616591 (+ 240us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:03.616635 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:03.616652 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:03.616753 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:03.616978 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 19:09:03.617012 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":115,"threads_started":1}
I20260925 19:09:03.618710 787 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:09:03.624220 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:03.620342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:57859 (local address 127.0.0.1:35699)
0925 19:09:03.620387 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:03.620389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:03.620505 (+ 116us) server_negotiation.cc:407] Connection header received
0925 19:09:03.620559 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:03.620560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:03.620571 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:03.620628 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:03.620760 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:03.621515 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:03.622662 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:03.623005 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:03.623071 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:03.623925 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:03.623927 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:03.623927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:03.623934 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:03.623952 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:03.623957 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:03.624014 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:03.624112 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 19:09:03.624127 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:09:03.626256 21849 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:03.626905 21849 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:03.634929 21849 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b7ad57fe7a243a29cf1f549d9f2c885: Generated new TSK 0
I20260925 19:09:03.635195 21849 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:09:03.642122 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 21773
W20260925 19:09:03.646407 21758 connection.cc:535] client connection to 127.0.0.1:35699 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35699): Connection reset by peer (error 104)
2026-09-25T19:09:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:09:03 dist-test-slave-rn98 krb5kdc[21872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:09:03 dist-test-slave-rn98 krb5kdc[21872](info): set up 2 sockets
Sep 25 19:09:03 dist-test-slave-rn98 krb5kdc[21872](info): commencing operation
krb5kdc: starting...
W20260925 19:09:04.758937 787 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:09:04 dist-test-slave-rn98 krb5kdc[21872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:09:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:09:04.886274 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36209
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:04.969777 21885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:04.970011 21885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:04.970053 21885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:04.971415 21885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:04.971462 21885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:04.971477 21885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:04.971489 21885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:04.971503 21885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:04.973500 21885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:04.973824 21885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:04.974079 21885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:04.979125 21890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:04.979126 21893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:04.979140 21891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:04.979400 21885 server_base.cc:1034] running on GCE node
I20260925 19:09:04.979677 21885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:04.979940 21885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:04.981105 21885 hybrid_clock.cc:648] HybridClock initialized: now 1790363344981083 us; error 37 us; skew 500 ppm
Sep 25 19:09:04 dist-test-slave-rn98 krb5kdc[21872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363344, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:04.983814 21885 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:09:04.984372 21885 webserver.cc:460] Webserver started at http://127.0.0.1:36879/ using document root <none> and password file <none>
I20260925 19:09:04.984602 21885 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:04.984649 21885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:04.984819 21885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:04.985893 21885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "395094ae4e5144c6a7f208a5372c6d80"
format_stamp: "Formatted at 2026-09-25 19:09:04 on dist-test-slave-rn98"
server_key: "23d4cc63f1cfcebd9b7d6d00bc146d16"
server_key_iv: "1f320a9bb16155a5e94320c5c2690f97"
server_key_version: "encryptionkey@0"
I20260925 19:09:04.986268 21885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "395094ae4e5144c6a7f208a5372c6d80"
format_stamp: "Formatted at 2026-09-25 19:09:04 on dist-test-slave-rn98"
server_key: "23d4cc63f1cfcebd9b7d6d00bc146d16"
server_key_iv: "1f320a9bb16155a5e94320c5c2690f97"
server_key_version: "encryptionkey@0"
I20260925 19:09:04.987753 21885 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:09:04.988608 21900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:04.988799 21885 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:09:04.988905 21885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "395094ae4e5144c6a7f208a5372c6d80"
format_stamp: "Formatted at 2026-09-25 19:09:04 on dist-test-slave-rn98"
server_key: "23d4cc63f1cfcebd9b7d6d00bc146d16"
server_key_iv: "1f320a9bb16155a5e94320c5c2690f97"
server_key_version: "encryptionkey@0"
I20260925 19:09:04.988991 21885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:04.992664 21885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:04.995048 21885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:04.995215 21885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:04.999337 21885 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36209
I20260925 19:09:04.999369 21952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36209 every 8 connection(s)
I20260925 19:09:04.999720 21885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:05.000792 21953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:05.001348 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 21885
I20260925 19:09:05.001505 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:09:05.001713 787 external_mini_cluster.cc:1428] Setting key 09fee649dbe5e497b157472a963e473c
I20260925 19:09:05.004132 21953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 395094ae4e5144c6a7f208a5372c6d80: Bootstrap starting.
I20260925 19:09:05.004437 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:05.002315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:44449 (local address 127.0.0.1:36209)
0925 19:09:05.002525 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:05.002533 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:05.002560 (+ 27us) server_negotiation.cc:407] Connection header received
0925 19:09:05.003066 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:05.003080 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:05.003122 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:05.003251 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:05.003974 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:05.003978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:05.003978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:05.003982 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:05.004032 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:05.004046 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:05.004151 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:05.004182 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 19:09:05.004262 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":108,"threads_started":1}
I20260925 19:09:05.004926 21953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 395094ae4e5144c6a7f208a5372c6d80: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:05.005264 21953 log.cc:824] T 00000000000000000000000000000000 P 395094ae4e5144c6a7f208a5372c6d80: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:05.005705 787 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:09:05.006157 21953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 395094ae4e5144c6a7f208a5372c6d80: No bootstrap required, opened a new log
I20260925 19:09:05.008469 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:05.007245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:58313 (local address 127.0.0.1:36209)
0925 19:09:05.007293 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:05.007294 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:05.007368 (+ 74us) server_negotiation.cc:407] Connection header received
0925 19:09:05.007440 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:05.007441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:05.007445 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:05.007486 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:05.008187 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:05.008188 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:05.008188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:05.008192 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:05.008205 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:05.008208 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:05.008256 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:05.008342 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 19:09:05.008354 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 19:09:05.008836 21953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 395094ae4e5144c6a7f208a5372c6d80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395094ae4e5144c6a7f208a5372c6d80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36209 } }
I20260925 19:09:05.009037 21953 raft_consensus.cc:374] T 00000000000000000000000000000000 P 395094ae4e5144c6a7f208a5372c6d80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:05.009100 21953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 395094ae4e5144c6a7f208a5372c6d80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395094ae4e5144c6a7f208a5372c6d80, State: Initialized, Role: FOLLOWER
I20260925 19:09:05.009222 21953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 395094ae4e5144c6a7f208a5372c6d80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395094ae4e5144c6a7f208a5372c6d80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36209 } }
I20260925 19:09:05.009303 21953 raft_consensus.cc:388] T 00000000000000000000000000000000 P 395094ae4e5144c6a7f208a5372c6d80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:05.009327 21953 raft_consensus.cc:482] T 00000000000000000000000000000000 P 395094ae4e5144c6a7f208a5372c6d80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:05.009400 21953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 395094ae4e5144c6a7f208a5372c6d80 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:05.009989 21953 raft_consensus.cc:504] T 00000000000000000000000000000000 P 395094ae4e5144c6a7f208a5372c6d80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395094ae4e5144c6a7f208a5372c6d80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36209 } }
I20260925 19:09:05.010130 21953 leader_election.cc:302] T 00000000000000000000000000000000 P 395094ae4e5144c6a7f208a5372c6d80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 395094ae4e5144c6a7f208a5372c6d80; no voters:
I20260925 19:09:05.010324 21953 leader_election.cc:288] T 00000000000000000000000000000000 P 395094ae4e5144c6a7f208a5372c6d80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:05.010409 21963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 395094ae4e5144c6a7f208a5372c6d80 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:05.010591 21963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 395094ae4e5144c6a7f208a5372c6d80 [term 1 LEADER]: Becoming Leader. State: Replica: 395094ae4e5144c6a7f208a5372c6d80, State: Running, Role: LEADER
I20260925 19:09:05.010622 21953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 395094ae4e5144c6a7f208a5372c6d80 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:05.010772 21963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 395094ae4e5144c6a7f208a5372c6d80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395094ae4e5144c6a7f208a5372c6d80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36209 } }
I20260925 19:09:05.011150 21965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 395094ae4e5144c6a7f208a5372c6d80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 395094ae4e5144c6a7f208a5372c6d80. Latest consensus state: current_term: 1 leader_uuid: "395094ae4e5144c6a7f208a5372c6d80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395094ae4e5144c6a7f208a5372c6d80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36209 } } }
I20260925 19:09:05.011209 21964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 395094ae4e5144c6a7f208a5372c6d80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "395094ae4e5144c6a7f208a5372c6d80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395094ae4e5144c6a7f208a5372c6d80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36209 } } }
I20260925 19:09:05.011262 21965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 395094ae4e5144c6a7f208a5372c6d80 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:05.011323 21964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 395094ae4e5144c6a7f208a5372c6d80 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:05.011603 21971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:05.012166 21971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:05.015921 21971 catalog_manager.cc:1347] Generated new cluster ID: b2e7137f688d4819afe7274f8775b5d2
I20260925 19:09:05.015978 21971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:09:05.048174 21971 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:05.048378 21971 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:05.054488 21971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 395094ae4e5144c6a7f208a5372c6d80: Generated new TSK 0
I20260925 19:09:05.054752 21971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:09:05.057225 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 21885
2026-09-25T19:09:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:09:05 dist-test-slave-rn98 krb5kdc[21984](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:09:05 dist-test-slave-rn98 krb5kdc[21984](info): set up 2 sockets
Sep 25 19:09:05 dist-test-slave-rn98 krb5kdc[21984](info): commencing operation
krb5kdc: starting...
W20260925 19:09:06.159129 787 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:09:06 dist-test-slave-rn98 krb5kdc[21984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:09:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:09:06.286906 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38737
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:06.371397 21997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:06.371636 21997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:06.371663 21997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:06.372900 21997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:06.372956 21997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:06.372970 21997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:06.372984 21997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:06.372996 21997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:06.374727 21997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38737
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:06.374981 21997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:06.375178 21997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:06.380656 22002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:06.380645 22003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:06.380650 22005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:06.380920 21997 server_base.cc:1034] running on GCE node
I20260925 19:09:06.381196 21997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:06.381582 21997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:06.382771 21997 hybrid_clock.cc:648] HybridClock initialized: now 1790363346382739 us; error 50 us; skew 500 ppm
Sep 25 19:09:06 dist-test-slave-rn98 krb5kdc[21984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:06.388857 21997 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:09:06.389366 21997 webserver.cc:460] Webserver started at http://127.0.0.1:44891/ using document root <none> and password file <none>
I20260925 19:09:06.389571 21997 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:06.389616 21997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:06.389729 21997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:06.390945 21997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "0bbbe3612ef34f3589670600f2480714"
format_stamp: "Formatted at 2026-09-25 19:09:06 on dist-test-slave-rn98"
server_key: "542472c4a49d1373b72c3e4fd1e8351c"
server_key_iv: "4e73fb77c05a26375d2a71f4bafad1c7"
server_key_version: "encryptionkey@0"
I20260925 19:09:06.391283 21997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "0bbbe3612ef34f3589670600f2480714"
format_stamp: "Formatted at 2026-09-25 19:09:06 on dist-test-slave-rn98"
server_key: "542472c4a49d1373b72c3e4fd1e8351c"
server_key_iv: "4e73fb77c05a26375d2a71f4bafad1c7"
server_key_version: "encryptionkey@0"
I20260925 19:09:06.392741 21997 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:09:06.393498 22012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:06.393697 21997 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:06.393810 21997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "0bbbe3612ef34f3589670600f2480714"
format_stamp: "Formatted at 2026-09-25 19:09:06 on dist-test-slave-rn98"
server_key: "542472c4a49d1373b72c3e4fd1e8351c"
server_key_iv: "4e73fb77c05a26375d2a71f4bafad1c7"
server_key_version: "encryptionkey@0"
I20260925 19:09:06.393893 21997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:06.404680 21997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:06.407406 21997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:06.407603 21997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:06.412086 21997 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38737
I20260925 19:09:06.412141 22064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38737 every 8 connection(s)
I20260925 19:09:06.412521 21997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:06.413537 22065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:06.416361 22065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bbbe3612ef34f3589670600f2480714: Bootstrap starting.
I20260925 19:09:06.417097 22065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bbbe3612ef34f3589670600f2480714: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:06.417421 22065 log.cc:824] T 00000000000000000000000000000000 P 0bbbe3612ef34f3589670600f2480714: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:06.418246 22065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bbbe3612ef34f3589670600f2480714: No bootstrap required, opened a new log
I20260925 19:09:06.420069 22065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bbbe3612ef34f3589670600f2480714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bbbe3612ef34f3589670600f2480714" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38737 } }
I20260925 19:09:06.420252 22065 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bbbe3612ef34f3589670600f2480714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:06.420279 22065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bbbe3612ef34f3589670600f2480714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bbbe3612ef34f3589670600f2480714, State: Initialized, Role: FOLLOWER
I20260925 19:09:06.420486 22065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bbbe3612ef34f3589670600f2480714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bbbe3612ef34f3589670600f2480714" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38737 } }
I20260925 19:09:06.420603 22065 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bbbe3612ef34f3589670600f2480714 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:06.420688 22065 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bbbe3612ef34f3589670600f2480714 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:06.420745 22065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bbbe3612ef34f3589670600f2480714 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:06.421355 22065 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bbbe3612ef34f3589670600f2480714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bbbe3612ef34f3589670600f2480714" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38737 } }
I20260925 19:09:06.421478 22065 leader_election.cc:302] T 00000000000000000000000000000000 P 0bbbe3612ef34f3589670600f2480714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bbbe3612ef34f3589670600f2480714; no voters:
I20260925 19:09:06.421706 22065 leader_election.cc:288] T 00000000000000000000000000000000 P 0bbbe3612ef34f3589670600f2480714 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:06.421792 22068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bbbe3612ef34f3589670600f2480714 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:06.421996 22065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bbbe3612ef34f3589670600f2480714 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:06.422297 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 21997
I20260925 19:09:06.422446 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:09:06.422700 787 external_mini_cluster.cc:1428] Setting key 7e0e58ee8eb739599d061465fbc21f36
I20260925 19:09:06.422000 22068 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bbbe3612ef34f3589670600f2480714 [term 1 LEADER]: Becoming Leader. State: Replica: 0bbbe3612ef34f3589670600f2480714, State: Running, Role: LEADER
I20260925 19:09:06.422823 22068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bbbe3612ef34f3589670600f2480714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bbbe3612ef34f3589670600f2480714" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38737 } }
I20260925 19:09:06.423521 22070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bbbe3612ef34f3589670600f2480714 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bbbe3612ef34f3589670600f2480714. Latest consensus state: current_term: 1 leader_uuid: "0bbbe3612ef34f3589670600f2480714" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bbbe3612ef34f3589670600f2480714" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38737 } } }
I20260925 19:09:06.423525 22069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bbbe3612ef34f3589670600f2480714 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bbbe3612ef34f3589670600f2480714" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bbbe3612ef34f3589670600f2480714" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38737 } } }
I20260925 19:09:06.423636 22070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bbbe3612ef34f3589670600f2480714 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:06.423645 22069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bbbe3612ef34f3589670600f2480714 [sys.catalog]: This master's current role is: LEADER
W20260925 19:09:06.424451 22084 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bbbe3612ef34f3589670600f2480714: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:09:06.424511 22084 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:09:06.424633 22085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:06.424793 22085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:06.426493 22085 catalog_manager.cc:1347] Generated new cluster ID: d810c6b7d4cd474789babbaeb09cdc6a
I20260925 19:09:06.426573 22085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:09:06 dist-test-slave-rn98 krb5kdc[21984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 19:09:06.434370 22083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:06.423416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:37139 (local address 127.0.0.1:38737)
0925 19:09:06.423825 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:06.423836 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:06.423871 (+ 35us) server_negotiation.cc:407] Connection header received
0925 19:09:06.424100 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:06.424110 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:06.424193 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:06.424381 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:06.425172 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:06.426443 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:06.427285 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:06.427740 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:06.427865 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:06.430546 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:06.430553 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:06.430553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:06.430569 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:06.432577 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:06.433034 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:06.433039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:06.433039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:06.433184 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:06.433437 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:06.433438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:06.433438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:06.433731 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:06.433926 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:06.434017 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 19:09:06.434133 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":355,"thread_start_us":81,"threads_started":1}
I20260925 19:09:06.435560 787 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:09:06.442068 22085 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:06.442881 22085 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:06.445672 22083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:06.436992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:52105 (local address 127.0.0.1:38737)
0925 19:09:06.437031 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:06.437033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:06.438833 (+ 1800us) server_negotiation.cc:407] Connection header received
0925 19:09:06.438912 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:06.438912 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:06.438923 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:06.438969 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:06.439618 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:06.440158 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:06.440892 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:06.441120 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:06.441232 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:06.442897 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:06.442899 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:06.442899 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:06.442906 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:06.444433 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:06.444830 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:06.444831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:06.444832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:06.444893 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:06.445126 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:06.445127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:06.445128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:06.445250 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:06.445379 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:06.445497 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 19:09:06.445563 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:09:06.448649 22085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bbbe3612ef34f3589670600f2480714: Generated new TSK 0
I20260925 19:09:06.448860 22085 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:09:06.456664 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 21997
W20260925 19:09:06.460974 21982 connection.cc:535] client connection to 127.0.0.1:38737 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38737): Connection reset by peer (error 104)
2026-09-25T19:09:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1398 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:09:06 dist-test-slave-rn98 krb5kdc[22096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:09:06 dist-test-slave-rn98 krb5kdc[22096](info): set up 2 sockets
Sep 25 19:09:06 dist-test-slave-rn98 krb5kdc[22096](info): commencing operation
krb5kdc: starting...
W20260925 19:09:07.506860 787 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:09:07 dist-test-slave-rn98 krb5kdc[22096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:09:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:09:07.634390 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45277
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:07.717002 22109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:07.717288 22109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:07.717331 22109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:07.718742 22109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:07.718837 22109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:07.718860 22109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:07.718880 22109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:07.718910 22109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:07.720535 22109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45277
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:07.720857 22109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:07.721108 22109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:07.726423 22117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:07.726423 22115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:07.726423 22114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:07.726751 22109 server_base.cc:1034] running on GCE node
I20260925 19:09:07.727116 22109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:07.727386 22109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:07.728538 22109 hybrid_clock.cc:648] HybridClock initialized: now 1790363347728525 us; error 36 us; skew 500 ppm
Sep 25 19:09:07 dist-test-slave-rn98 krb5kdc[22096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:07.731376 22109 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:09:07.731946 22109 webserver.cc:460] Webserver started at http://127.0.0.1:44563/ using document root <none> and password file <none>
I20260925 19:09:07.732197 22109 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:07.732275 22109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:07.732424 22109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:07.733549 22109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "062dccd90bc2464680cdd2264c4cb5a4"
format_stamp: "Formatted at 2026-09-25 19:09:07 on dist-test-slave-rn98"
server_key: "f166b1e3641e4b4ac27e5f424a573273"
server_key_iv: "365095fa1d8ea977f556dcdc9a4f9e97"
server_key_version: "encryptionkey@0"
I20260925 19:09:07.733932 22109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "062dccd90bc2464680cdd2264c4cb5a4"
format_stamp: "Formatted at 2026-09-25 19:09:07 on dist-test-slave-rn98"
server_key: "f166b1e3641e4b4ac27e5f424a573273"
server_key_iv: "365095fa1d8ea977f556dcdc9a4f9e97"
server_key_version: "encryptionkey@0"
I20260925 19:09:07.735513 22109 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:09:07.736485 22124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:07.736729 22109 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:07.736848 22109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "062dccd90bc2464680cdd2264c4cb5a4"
format_stamp: "Formatted at 2026-09-25 19:09:07 on dist-test-slave-rn98"
server_key: "f166b1e3641e4b4ac27e5f424a573273"
server_key_iv: "365095fa1d8ea977f556dcdc9a4f9e97"
server_key_version: "encryptionkey@0"
I20260925 19:09:07.736936 22109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:07.754920 22109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:07.757392 22109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:07.757575 22109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:07.762145 22109 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45277
I20260925 19:09:07.762233 22176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45277 every 8 connection(s)
I20260925 19:09:07.762636 22109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:07.763716 22177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:07.766392 22177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 062dccd90bc2464680cdd2264c4cb5a4: Bootstrap starting.
I20260925 19:09:07.767113 22177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 062dccd90bc2464680cdd2264c4cb5a4: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:07.767445 22177 log.cc:824] T 00000000000000000000000000000000 P 062dccd90bc2464680cdd2264c4cb5a4: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:07.768272 22177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 062dccd90bc2464680cdd2264c4cb5a4: No bootstrap required, opened a new log
I20260925 19:09:07.769852 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 22109
I20260925 19:09:07.769968 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:09:07.770223 787 external_mini_cluster.cc:1428] Setting key db4c9bc94e346160e8547568607d1859
I20260925 19:09:07.770291 22177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 062dccd90bc2464680cdd2264c4cb5a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "062dccd90bc2464680cdd2264c4cb5a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45277 } }
I20260925 19:09:07.770491 22177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 062dccd90bc2464680cdd2264c4cb5a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:07.770545 22177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 062dccd90bc2464680cdd2264c4cb5a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 062dccd90bc2464680cdd2264c4cb5a4, State: Initialized, Role: FOLLOWER
I20260925 19:09:07.770718 22177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 062dccd90bc2464680cdd2264c4cb5a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "062dccd90bc2464680cdd2264c4cb5a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45277 } }
I20260925 19:09:07.770900 22177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 062dccd90bc2464680cdd2264c4cb5a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:07.770989 22177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 062dccd90bc2464680cdd2264c4cb5a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:07.771078 22177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 062dccd90bc2464680cdd2264c4cb5a4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:07.772104 22177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 062dccd90bc2464680cdd2264c4cb5a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "062dccd90bc2464680cdd2264c4cb5a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45277 } }
I20260925 19:09:07.772207 22177 leader_election.cc:302] T 00000000000000000000000000000000 P 062dccd90bc2464680cdd2264c4cb5a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 062dccd90bc2464680cdd2264c4cb5a4; no voters:
I20260925 19:09:07.772423 22177 leader_election.cc:288] T 00000000000000000000000000000000 P 062dccd90bc2464680cdd2264c4cb5a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:07.772477 22182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 062dccd90bc2464680cdd2264c4cb5a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:07.772655 22182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 062dccd90bc2464680cdd2264c4cb5a4 [term 1 LEADER]: Becoming Leader. State: Replica: 062dccd90bc2464680cdd2264c4cb5a4, State: Running, Role: LEADER
I20260925 19:09:07.772755 22177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 062dccd90bc2464680cdd2264c4cb5a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:07.772804 22182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 062dccd90bc2464680cdd2264c4cb5a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "062dccd90bc2464680cdd2264c4cb5a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45277 } }
I20260925 19:09:07.773201 22182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 062dccd90bc2464680cdd2264c4cb5a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 062dccd90bc2464680cdd2264c4cb5a4. Latest consensus state: current_term: 1 leader_uuid: "062dccd90bc2464680cdd2264c4cb5a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "062dccd90bc2464680cdd2264c4cb5a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45277 } } }
I20260925 19:09:07.773321 22182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 062dccd90bc2464680cdd2264c4cb5a4 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:07.773521 22183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 062dccd90bc2464680cdd2264c4cb5a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "062dccd90bc2464680cdd2264c4cb5a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "062dccd90bc2464680cdd2264c4cb5a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45277 } } }
I20260925 19:09:07.773865 22183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 062dccd90bc2464680cdd2264c4cb5a4 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:07.774775 22187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:07.775365 22187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:07.776536 22181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:07.770778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:48673 (local address 127.0.0.1:45277)
0925 19:09:07.771008 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:07.771019 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:07.771060 (+ 41us) server_negotiation.cc:407] Connection header received
0925 19:09:07.771215 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:07.771226 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:07.771273 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:07.771414 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:07.771611 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:07.772748 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:07.773928 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:07.774577 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:07.774753 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:07.775707 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:07.775711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:07.775711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:07.775942 (+ 231us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:07.775980 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:07.775994 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:07.776235 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:07.776289 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 19:09:07.776319 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":103,"threads_started":1}
I20260925 19:09:07.777077 22187 catalog_manager.cc:1347] Generated new cluster ID: 97f64a2ab5544bd6b0862eda2a3889b0
I20260925 19:09:07.777153 22187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:09:07.783169 22187 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:07.783381 787 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:09:07.783890 22187 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:07.789105 22181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:07.785015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:46877 (local address 127.0.0.1:45277)
0925 19:09:07.785059 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:07.785061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:07.785226 (+ 165us) server_negotiation.cc:407] Connection header received
0925 19:09:07.785281 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:07.785282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:07.785296 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:07.785356 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:07.785491 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:07.786332 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:07.787519 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:07.787874 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:07.787940 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:07.788794 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:07.788796 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:07.788796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:07.788802 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:07.788821 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:07.788827 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:07.788902 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:07.788987 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 19:09:07.789003 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:09:07.789369 22187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 062dccd90bc2464680cdd2264c4cb5a4: Generated new TSK 0
I20260925 19:09:07.789562 22187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:09:07.795171 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 22109
W20260925 19:09:07.799688 22094 connection.cc:535] client connection to 127.0.0.1:45277 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45277): Connection reset by peer (error 104)
2026-09-25T19:09:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:09:07 dist-test-slave-rn98 krb5kdc[22208](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:09:07 dist-test-slave-rn98 krb5kdc[22208](info): set up 2 sockets
Sep 25 19:09:07 dist-test-slave-rn98 krb5kdc[22208](info): commencing operation
krb5kdc: starting...
W20260925 19:09:08.838889 787 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:09:08 dist-test-slave-rn98 krb5kdc[22208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:09:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:09:08.965387 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45897
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:09.048588 22221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:09.048813 22221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:09.048839 22221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:09.050026 22221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:09.050071 22221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:09.050086 22221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:09.050098 22221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:09.050110 22221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:09.051844 22221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45897
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363087086945-787-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:09.052148 22221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:09.052403 22221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:09.057894 22229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:09.057866 22227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:09.057934 22221 server_base.cc:1034] running on GCE node
W20260925 19:09:09.057865 22226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:09.058495 22221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:09.058879 22221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:09.060079 22221 hybrid_clock.cc:648] HybridClock initialized: now 1790363349060053 us; error 50 us; skew 500 ppm
Sep 25 19:09:09 dist-test-slave-rn98 krb5kdc[22208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:09.062958 22221 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:09:09.063500 22221 webserver.cc:460] Webserver started at http://127.0.0.1:42551/ using document root <none> and password file <none>
I20260925 19:09:09.063773 22221 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:09.063849 22221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:09.064002 22221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:09.065122 22221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "e6c7a63a9caa46998ac919a8175b1ced"
format_stamp: "Formatted at 2026-09-25 19:09:09 on dist-test-slave-rn98"
server_key: "879ed5aa3dab9bb1bc2acbb00b8a92b2"
server_key_iv: "c6624e32fe48151a49c90fe829a53be8"
server_key_version: "encryptionkey@0"
I20260925 19:09:09.065505 22221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "e6c7a63a9caa46998ac919a8175b1ced"
format_stamp: "Formatted at 2026-09-25 19:09:09 on dist-test-slave-rn98"
server_key: "879ed5aa3dab9bb1bc2acbb00b8a92b2"
server_key_iv: "c6624e32fe48151a49c90fe829a53be8"
server_key_version: "encryptionkey@0"
I20260925 19:09:09.067013 22221 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:09:09.067943 22236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:09.068198 22221 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:09:09.068310 22221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "e6c7a63a9caa46998ac919a8175b1ced"
format_stamp: "Formatted at 2026-09-25 19:09:09 on dist-test-slave-rn98"
server_key: "879ed5aa3dab9bb1bc2acbb00b8a92b2"
server_key_iv: "c6624e32fe48151a49c90fe829a53be8"
server_key_version: "encryptionkey@0"
I20260925 19:09:09.068393 22221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:09.072192 22221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:09.074736 22221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:09.074914 22221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:09.079568 22221 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45897
I20260925 19:09:09.079602 22288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45897 every 8 connection(s)
I20260925 19:09:09.079984 22221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:09.080606 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 22221
I20260925 19:09:09.080744 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:09:09.080938 787 external_mini_cluster.cc:1428] Setting key adb4ff801781b19b9600e19a21a0b898
I20260925 19:09:09.081277 22289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:09.084020 22292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:09.081552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:42299 (local address 127.0.0.1:45897)
0925 19:09:09.081728 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:09.081743 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:09.081769 (+ 26us) server_negotiation.cc:407] Connection header received
0925 19:09:09.082525 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:09.082534 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:09.082567 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:09.082796 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:09.083529 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:09.083532 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:09.083533 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:09.083537 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:09.083576 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:09.083588 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:09.083644 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:09.083737 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 19:09:09.083801 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260925 19:09:09.085196 22289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6c7a63a9caa46998ac919a8175b1ced: Bootstrap starting.
I20260925 19:09:09.085395 787 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:09:09.086025 22289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6c7a63a9caa46998ac919a8175b1ced: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:09.086364 22289 log.cc:824] T 00000000000000000000000000000000 P e6c7a63a9caa46998ac919a8175b1ced: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:09.087313 22289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6c7a63a9caa46998ac919a8175b1ced: No bootstrap required, opened a new log
I20260925 19:09:09.088367 22292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:09.087148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.225:35573 (local address 127.0.0.1:45897)
0925 19:09:09.087212 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:09.087214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:09.087332 (+ 118us) server_negotiation.cc:407] Connection header received
0925 19:09:09.087415 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:09.087416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:09.087421 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:09.087480 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:09.088118 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:09.088120 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:09.088120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:09.088124 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:09.088140 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:09.088149 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:09.088220 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:09.088282 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 19:09:09.088298 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 19:09:09.089663 22289 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6c7a63a9caa46998ac919a8175b1ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c7a63a9caa46998ac919a8175b1ced" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45897 } }
I20260925 19:09:09.089813 22289 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6c7a63a9caa46998ac919a8175b1ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:09.089836 22289 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6c7a63a9caa46998ac919a8175b1ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6c7a63a9caa46998ac919a8175b1ced, State: Initialized, Role: FOLLOWER
I20260925 19:09:09.090001 22289 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6c7a63a9caa46998ac919a8175b1ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c7a63a9caa46998ac919a8175b1ced" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45897 } }
I20260925 19:09:09.090082 22289 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6c7a63a9caa46998ac919a8175b1ced [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:09.090158 22289 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6c7a63a9caa46998ac919a8175b1ced [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:09.090210 22289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6c7a63a9caa46998ac919a8175b1ced [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:09.090904 22289 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6c7a63a9caa46998ac919a8175b1ced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c7a63a9caa46998ac919a8175b1ced" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45897 } }
I20260925 19:09:09.091007 22289 leader_election.cc:302] T 00000000000000000000000000000000 P e6c7a63a9caa46998ac919a8175b1ced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6c7a63a9caa46998ac919a8175b1ced; no voters:
I20260925 19:09:09.091251 22289 leader_election.cc:288] T 00000000000000000000000000000000 P e6c7a63a9caa46998ac919a8175b1ced [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:09.091491 22300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6c7a63a9caa46998ac919a8175b1ced [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:09.091552 22289 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6c7a63a9caa46998ac919a8175b1ced [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:09.091652 22300 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6c7a63a9caa46998ac919a8175b1ced [term 1 LEADER]: Becoming Leader. State: Replica: e6c7a63a9caa46998ac919a8175b1ced, State: Running, Role: LEADER
I20260925 19:09:09.091768 22300 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6c7a63a9caa46998ac919a8175b1ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c7a63a9caa46998ac919a8175b1ced" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45897 } }
I20260925 19:09:09.092151 22300 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6c7a63a9caa46998ac919a8175b1ced [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6c7a63a9caa46998ac919a8175b1ced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c7a63a9caa46998ac919a8175b1ced" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45897 } } }
I20260925 19:09:09.092237 22299 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6c7a63a9caa46998ac919a8175b1ced [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6c7a63a9caa46998ac919a8175b1ced. Latest consensus state: current_term: 1 leader_uuid: "e6c7a63a9caa46998ac919a8175b1ced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c7a63a9caa46998ac919a8175b1ced" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45897 } } }
I20260925 19:09:09.092314 22300 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6c7a63a9caa46998ac919a8175b1ced [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:09.092422 22299 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6c7a63a9caa46998ac919a8175b1ced [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:09.092621 22308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:09.093214 22308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:09.102757 22308 catalog_manager.cc:1347] Generated new cluster ID: ee70eedf52f94774b505d0089a653dab
I20260925 19:09:09.102870 22308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:09:09.111270 22308 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:09.111481 22308 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:09.118178 22308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6c7a63a9caa46998ac919a8175b1ced: Generated new TSK 0
I20260925 19:09:09.118404 22308 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:09:09.120414 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 22221
2026-09-25T19:09:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:09:09 dist-test-slave-rn98 krb5kdc[22320](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:09:09 dist-test-slave-rn98 krb5kdc[22320](info): set up 2 sockets
Sep 25 19:09:09 dist-test-slave-rn98 krb5kdc[22320](info): commencing operation
krb5kdc: starting...
W20260925 19:09:10.159000 787 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:09:10 dist-test-slave-rn98 krb5kdc[22320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:09:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:09:10.286365 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:40973
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:43937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:40973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:10.370710 22333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:10.371021 22333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:10.371050 22333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:10.372272 22333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:10.372318 22333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:10.372331 22333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:10.372344 22333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:10.372356 22333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:10.373865 22333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:43937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:40973
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:40973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:10.374087 22333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:10.374271 22333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:10.379631 22338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:10.379892 22333 server_base.cc:1034] running on GCE node
W20260925 19:09:10.379635 22339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:10.379631 22341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:10.380275 22333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:10.380602 22333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:10.381788 22333 hybrid_clock.cc:648] HybridClock initialized: now 1790363350381776 us; error 52 us; skew 500 ppm
Sep 25 19:09:10 dist-test-slave-rn98 krb5kdc[22320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:10.384620 22333 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:09:10.385200 22333 webserver.cc:460] Webserver started at http://127.0.196.254:39713/ using document root <none> and password file <none>
I20260925 19:09:10.385439 22333 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:10.385511 22333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:10.385665 22333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:10.386886 22333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "bf804e0a175e406a99b0065c1915e6f9"
format_stamp: "Formatted at 2026-09-25 19:09:10 on dist-test-slave-rn98"
server_key: "48deae4a963de37f8a5cf3cb78284147"
server_key_iv: "ab92cc162730fe087bc75fc554c2d6c0"
server_key_version: "encryptionkey@0"
I20260925 19:09:10.387270 22333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "bf804e0a175e406a99b0065c1915e6f9"
format_stamp: "Formatted at 2026-09-25 19:09:10 on dist-test-slave-rn98"
server_key: "48deae4a963de37f8a5cf3cb78284147"
server_key_iv: "ab92cc162730fe087bc75fc554c2d6c0"
server_key_version: "encryptionkey@0"
I20260925 19:09:10.388731 22333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:09:10.389623 22348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:10.389808 22333 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:10.389933 22333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "bf804e0a175e406a99b0065c1915e6f9"
format_stamp: "Formatted at 2026-09-25 19:09:10 on dist-test-slave-rn98"
server_key: "48deae4a963de37f8a5cf3cb78284147"
server_key_iv: "ab92cc162730fe087bc75fc554c2d6c0"
server_key_version: "encryptionkey@0"
I20260925 19:09:10.390020 22333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:10.409662 22333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:10.412235 22333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:10.412416 22333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:10.416855 22333 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:40973
I20260925 19:09:10.417052 22400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:40973 every 8 connection(s)
I20260925 19:09:10.417318 22333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:10.418326 22401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:10.421046 22401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf804e0a175e406a99b0065c1915e6f9: Bootstrap starting.
I20260925 19:09:10.421769 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 22333
I20260925 19:09:10.421900 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:09:10.421918 22401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf804e0a175e406a99b0065c1915e6f9: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:10.422137 787 external_mini_cluster.cc:1428] Setting key 62f48460bc17c955a076d9e152026b6d
I20260925 19:09:10.422284 22401 log.cc:824] T 00000000000000000000000000000000 P bf804e0a175e406a99b0065c1915e6f9: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:10.423285 22401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf804e0a175e406a99b0065c1915e6f9: No bootstrap required, opened a new log
I20260925 19:09:10.425953 22401 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf804e0a175e406a99b0065c1915e6f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf804e0a175e406a99b0065c1915e6f9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 40973 } }
I20260925 19:09:10.426115 22401 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf804e0a175e406a99b0065c1915e6f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:10.426139 22401 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf804e0a175e406a99b0065c1915e6f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf804e0a175e406a99b0065c1915e6f9, State: Initialized, Role: FOLLOWER
I20260925 19:09:10.426301 22401 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf804e0a175e406a99b0065c1915e6f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf804e0a175e406a99b0065c1915e6f9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 40973 } }
I20260925 19:09:10.426379 22401 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf804e0a175e406a99b0065c1915e6f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:10.426430 22401 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf804e0a175e406a99b0065c1915e6f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:10.426489 22401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf804e0a175e406a99b0065c1915e6f9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:10.427208 22401 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf804e0a175e406a99b0065c1915e6f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf804e0a175e406a99b0065c1915e6f9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 40973 } }
I20260925 19:09:10.427309 22401 leader_election.cc:302] T 00000000000000000000000000000000 P bf804e0a175e406a99b0065c1915e6f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf804e0a175e406a99b0065c1915e6f9; no voters:
I20260925 19:09:10.427567 22401 leader_election.cc:288] T 00000000000000000000000000000000 P bf804e0a175e406a99b0065c1915e6f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:10.427615 22406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf804e0a175e406a99b0065c1915e6f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:10.427811 22406 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf804e0a175e406a99b0065c1915e6f9 [term 1 LEADER]: Becoming Leader. State: Replica: bf804e0a175e406a99b0065c1915e6f9, State: Running, Role: LEADER
I20260925 19:09:10.427870 22401 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf804e0a175e406a99b0065c1915e6f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:10.427958 22406 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf804e0a175e406a99b0065c1915e6f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf804e0a175e406a99b0065c1915e6f9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 40973 } }
I20260925 19:09:10.428448 22408 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf804e0a175e406a99b0065c1915e6f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf804e0a175e406a99b0065c1915e6f9. Latest consensus state: current_term: 1 leader_uuid: "bf804e0a175e406a99b0065c1915e6f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf804e0a175e406a99b0065c1915e6f9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 40973 } } }
I20260925 19:09:10.428576 22408 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf804e0a175e406a99b0065c1915e6f9 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:10.428831 22415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:10.429361 22415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:10.429338 22407 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf804e0a175e406a99b0065c1915e6f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf804e0a175e406a99b0065c1915e6f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf804e0a175e406a99b0065c1915e6f9" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 40973 } } }
I20260925 19:09:10.429453 22407 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf804e0a175e406a99b0065c1915e6f9 [sys.catalog]: This master's current role is: LEADER
Sep 25 19:09:10 dist-test-slave-rn98 krb5kdc[22320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:09:10.431794 22415 catalog_manager.cc:1347] Generated new cluster ID: ae79acd5e24043aaa8f7c75b24c8891d
I20260925 19:09:10.431839 22415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:09:10.436825 22405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:10.422774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34463 (local address 127.0.196.254:40973)
0925 19:09:10.422973 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:10.422983 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:10.423025 (+ 42us) server_negotiation.cc:407] Connection header received
0925 19:09:10.424664 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:10.424671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:10.424727 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:10.424852 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:10.425787 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:10.427260 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:10.428374 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:10.429007 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:10.429133 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:10.432394 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:10.432398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:10.432398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:10.432410 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:10.434711 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:10.435221 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:10.435225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:10.435225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:10.435346 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:10.435606 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:10.435607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:10.435608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:10.436131 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:10.436307 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:10.436435 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 19:09:10.436558 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260925 19:09:10.438063 787 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:09:10.444308 22415 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:10.445035 22415 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:10.450480 22405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:10.442749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49281 (local address 127.0.196.254:40973)
0925 19:09:10.442791 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:10.442793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:10.442988 (+ 195us) server_negotiation.cc:407] Connection header received
0925 19:09:10.443031 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:10.443033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:10.443045 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:10.443112 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:10.443833 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:10.444596 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:10.445798 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:10.446048 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:10.446131 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:10.447525 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:10.447526 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:10.447526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:10.447534 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:10.449026 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:10.449417 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:10.449418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:10.449419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:10.449486 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:10.449718 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:10.449719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:10.449719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:10.449833 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:10.449970 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:10.450275 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 19:09:10.450345 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:09:10.457820 22415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf804e0a175e406a99b0065c1915e6f9: Generated new TSK 0
I20260925 19:09:10.458061 22415 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:09:10.465202 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 22333
W20260925 19:09:10.469574 22318 connection.cc:535] client connection to 127.0.196.254:40973 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.254:40973): Connection reset by peer (error 104)
2026-09-25T19:09:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:09:10 dist-test-slave-rn98 krb5kdc[22432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:09:10 dist-test-slave-rn98 krb5kdc[22432](info): set up 2 sockets
Sep 25 19:09:10 dist-test-slave-rn98 krb5kdc[22432](info): commencing operation
krb5kdc: starting...
W20260925 19:09:11.579018 787 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:09:11 dist-test-slave-rn98 krb5kdc[22432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:09:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:09:11.706218 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:46049
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:35091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:46049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:11.788990 22445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:11.789288 22445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:11.789330 22445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:11.790653 22445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:11.790747 22445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:11.790774 22445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:11.790794 22445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:11.790863 22445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:11.792533 22445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:35091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:46049
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:46049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:11.792881 22445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:11.793145 22445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:09:11.798458 22445 server_base.cc:1034] running on GCE node
W20260925 19:09:11.798424 22453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:11.798419 22450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:11.798449 22451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:11.798990 22445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:11.799335 22445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:11.800562 22445 hybrid_clock.cc:648] HybridClock initialized: now 1790363351800529 us; error 42 us; skew 500 ppm
Sep 25 19:09:11 dist-test-slave-rn98 krb5kdc[22432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:11.807500 22445 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:09:11.808130 22445 webserver.cc:460] Webserver started at http://127.0.196.254:39373/ using document root <none> and password file <none>
I20260925 19:09:11.808399 22445 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:11.808480 22445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:11.808647 22445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:11.809791 22445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "ebf271ef63704028aba798b480a6af6b"
format_stamp: "Formatted at 2026-09-25 19:09:11 on dist-test-slave-rn98"
server_key: "15296f2df2cbb5c95b4befcab0494722"
server_key_iv: "575902a86878ff80aac1323c5e34454b"
server_key_version: "encryptionkey@0"
I20260925 19:09:11.810220 22445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "ebf271ef63704028aba798b480a6af6b"
format_stamp: "Formatted at 2026-09-25 19:09:11 on dist-test-slave-rn98"
server_key: "15296f2df2cbb5c95b4befcab0494722"
server_key_iv: "575902a86878ff80aac1323c5e34454b"
server_key_version: "encryptionkey@0"
I20260925 19:09:11.811837 22445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:09:11.812799 22460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:11.813016 22445 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:09:11.813138 22445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "ebf271ef63704028aba798b480a6af6b"
format_stamp: "Formatted at 2026-09-25 19:09:11 on dist-test-slave-rn98"
server_key: "15296f2df2cbb5c95b4befcab0494722"
server_key_iv: "575902a86878ff80aac1323c5e34454b"
server_key_version: "encryptionkey@0"
I20260925 19:09:11.813228 22445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:11.830178 22445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:11.832858 22445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:11.833063 22445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:11.837330 22445 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:46049
I20260925 19:09:11.837371 22512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:46049 every 8 connection(s)
I20260925 19:09:11.837774 22445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:11.838874 22513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:11.841320 22513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebf271ef63704028aba798b480a6af6b: Bootstrap starting.
I20260925 19:09:11.841770 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 22445
I20260925 19:09:11.841971 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:09:11.841988 22513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebf271ef63704028aba798b480a6af6b: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:11.842200 787 external_mini_cluster.cc:1428] Setting key 3f034507d8e19fe37161c5e09a636d08
I20260925 19:09:11.842342 22513 log.cc:824] T 00000000000000000000000000000000 P ebf271ef63704028aba798b480a6af6b: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:11.843353 22513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebf271ef63704028aba798b480a6af6b: No bootstrap required, opened a new log
I20260925 19:09:11.845974 22513 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebf271ef63704028aba798b480a6af6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf271ef63704028aba798b480a6af6b" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 46049 } }
I20260925 19:09:11.846199 22513 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebf271ef63704028aba798b480a6af6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:11.846233 22513 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebf271ef63704028aba798b480a6af6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebf271ef63704028aba798b480a6af6b, State: Initialized, Role: FOLLOWER
I20260925 19:09:11.846370 22513 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebf271ef63704028aba798b480a6af6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf271ef63704028aba798b480a6af6b" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 46049 } }
I20260925 19:09:11.846477 22513 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebf271ef63704028aba798b480a6af6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:11.846545 22513 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebf271ef63704028aba798b480a6af6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:11.846601 22513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebf271ef63704028aba798b480a6af6b [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:11.847409 22513 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebf271ef63704028aba798b480a6af6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf271ef63704028aba798b480a6af6b" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 46049 } }
I20260925 19:09:11.847568 22513 leader_election.cc:302] T 00000000000000000000000000000000 P ebf271ef63704028aba798b480a6af6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebf271ef63704028aba798b480a6af6b; no voters:
I20260925 19:09:11.847807 22513 leader_election.cc:288] T 00000000000000000000000000000000 P ebf271ef63704028aba798b480a6af6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:11.847939 22518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebf271ef63704028aba798b480a6af6b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:11.848134 22513 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebf271ef63704028aba798b480a6af6b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:11.848157 22518 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebf271ef63704028aba798b480a6af6b [term 1 LEADER]: Becoming Leader. State: Replica: ebf271ef63704028aba798b480a6af6b, State: Running, Role: LEADER
I20260925 19:09:11.848344 22518 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebf271ef63704028aba798b480a6af6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf271ef63704028aba798b480a6af6b" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 46049 } }
I20260925 19:09:11.848711 22519 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebf271ef63704028aba798b480a6af6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebf271ef63704028aba798b480a6af6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf271ef63704028aba798b480a6af6b" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 46049 } } }
I20260925 19:09:11.848796 22519 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebf271ef63704028aba798b480a6af6b [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:11.848966 22520 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebf271ef63704028aba798b480a6af6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebf271ef63704028aba798b480a6af6b. Latest consensus state: current_term: 1 leader_uuid: "ebf271ef63704028aba798b480a6af6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf271ef63704028aba798b480a6af6b" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 46049 } } }
I20260925 19:09:11.849020 22520 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebf271ef63704028aba798b480a6af6b [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:11.849063 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:11.842916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38279 (local address 127.0.196.254:46049)
0925 19:09:11.843097 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:11.843107 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:11.843142 (+ 35us) server_negotiation.cc:407] Connection header received
0925 19:09:11.843706 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:11.843715 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:11.843775 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:11.843915 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:11.844386 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:11.845315 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:11.846482 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:11.847053 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:11.847151 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:11.848091 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:11.848096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:11.848096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:11.848315 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:11.848374 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:11.848396 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:11.848481 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:11.848776 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 19:09:11.848811 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260925 19:09:11.849671 22527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:11.850162 787 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:09:11.850271 22527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:11.852196 22527 catalog_manager.cc:1347] Generated new cluster ID: 43eff68d645043a986b822c19dd42409
I20260925 19:09:11.852254 22527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:09:11.855684 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:11.851818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36061 (local address 127.0.196.254:46049)
0925 19:09:11.851858 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:11.851860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:11.851969 (+ 109us) server_negotiation.cc:407] Connection header received
0925 19:09:11.852018 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:11.852018 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:11.852029 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:11.852097 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:11.852264 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:11.853024 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:11.854119 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:11.854452 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:11.854496 (+ 44us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:11.855392 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:11.855394 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:11.855394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:11.855401 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:11.855418 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:11.855422 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:11.855456 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:11.855591 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 19:09:11.855603 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:09:11.877556 22527 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:11.878137 22527 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:11.888712 22527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebf271ef63704028aba798b480a6af6b: Generated new TSK 0
I20260925 19:09:11.888988 22527 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:09:11.891288 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 22445
W20260925 19:09:11.896031 22430 connection.cc:535] client connection to 127.0.196.254:46049 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.254:46049): Connection reset by peer (error 104)
2026-09-25T19:09:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:09:11 dist-test-slave-rn98 krb5kdc[22544](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:09:11 dist-test-slave-rn98 krb5kdc[22544](info): set up 2 sockets
Sep 25 19:09:11 dist-test-slave-rn98 krb5kdc[22544](info): commencing operation
krb5kdc: starting...
W20260925 19:09:12.971479 787 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:09:13 dist-test-slave-rn98 krb5kdc[22544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:09:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:09:13.098347 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:43041
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:35937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:43041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:13.185950 22557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:13.186228 22557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:13.186268 22557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:13.187505 22557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:13.187577 22557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:13.187602 22557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:13.187623 22557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:13.187652 22557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:13.189162 22557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:35937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:43041
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:43041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:13.189459 22557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:13.189697 22557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:13.194855 22562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:13.194960 22565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:13.194855 22563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:13.195132 22557 server_base.cc:1034] running on GCE node
I20260925 19:09:13.195394 22557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:13.195731 22557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:13.196928 22557 hybrid_clock.cc:648] HybridClock initialized: now 1790363353196899 us; error 36 us; skew 500 ppm
Sep 25 19:09:13 dist-test-slave-rn98 krb5kdc[22544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:13.199815 22557 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:09:13.200364 22557 webserver.cc:460] Webserver started at http://127.0.196.254:36919/ using document root <none> and password file <none>
I20260925 19:09:13.200625 22557 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:13.200702 22557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:13.200877 22557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:13.202085 22557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "e6917978d11d493f9c1861cdefdf81f2"
format_stamp: "Formatted at 2026-09-25 19:09:13 on dist-test-slave-rn98"
server_key: "31dbebf21de0ba6a1b03986d1a939af5"
server_key_iv: "422b3cca16c3b7b0ddc3c2a777ca1dae"
server_key_version: "encryptionkey@0"
I20260925 19:09:13.202499 22557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "e6917978d11d493f9c1861cdefdf81f2"
format_stamp: "Formatted at 2026-09-25 19:09:13 on dist-test-slave-rn98"
server_key: "31dbebf21de0ba6a1b03986d1a939af5"
server_key_iv: "422b3cca16c3b7b0ddc3c2a777ca1dae"
server_key_version: "encryptionkey@0"
I20260925 19:09:13.204092 22557 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:13.205116 22572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:13.205350 22557 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:13.205468 22557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "e6917978d11d493f9c1861cdefdf81f2"
format_stamp: "Formatted at 2026-09-25 19:09:13 on dist-test-slave-rn98"
server_key: "31dbebf21de0ba6a1b03986d1a939af5"
server_key_iv: "422b3cca16c3b7b0ddc3c2a777ca1dae"
server_key_version: "encryptionkey@0"
I20260925 19:09:13.205561 22557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:13.209118 22557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:13.211390 22557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:13.211552 22557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:13.215736 22557 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:43041
I20260925 19:09:13.215767 22624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:43041 every 8 connection(s)
I20260925 19:09:13.216116 22557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:13.217150 22625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:13.219587 22625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6917978d11d493f9c1861cdefdf81f2: Bootstrap starting.
I20260925 19:09:13.220187 22625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6917978d11d493f9c1861cdefdf81f2: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:13.220491 22625 log.cc:824] T 00000000000000000000000000000000 P e6917978d11d493f9c1861cdefdf81f2: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:13.221256 22625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6917978d11d493f9c1861cdefdf81f2: No bootstrap required, opened a new log
I20260925 19:09:13.222968 22625 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6917978d11d493f9c1861cdefdf81f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6917978d11d493f9c1861cdefdf81f2" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 43041 } }
I20260925 19:09:13.223121 22625 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6917978d11d493f9c1861cdefdf81f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:13.223145 22625 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6917978d11d493f9c1861cdefdf81f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6917978d11d493f9c1861cdefdf81f2, State: Initialized, Role: FOLLOWER
I20260925 19:09:13.223322 22625 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6917978d11d493f9c1861cdefdf81f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6917978d11d493f9c1861cdefdf81f2" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 43041 } }
I20260925 19:09:13.223387 22625 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6917978d11d493f9c1861cdefdf81f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:13.223445 22625 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6917978d11d493f9c1861cdefdf81f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:13.223515 22625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6917978d11d493f9c1861cdefdf81f2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:13.223779 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 22557
I20260925 19:09:13.223897 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:09:13.224077 787 external_mini_cluster.cc:1428] Setting key 1bf1c1d837ca90403129b24730b9b0df
I20260925 19:09:13.224253 22625 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6917978d11d493f9c1861cdefdf81f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6917978d11d493f9c1861cdefdf81f2" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 43041 } }
I20260925 19:09:13.224377 22625 leader_election.cc:302] T 00000000000000000000000000000000 P e6917978d11d493f9c1861cdefdf81f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6917978d11d493f9c1861cdefdf81f2; no voters:
I20260925 19:09:13.224581 22625 leader_election.cc:288] T 00000000000000000000000000000000 P e6917978d11d493f9c1861cdefdf81f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:13.224828 22628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6917978d11d493f9c1861cdefdf81f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:13.224912 22625 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6917978d11d493f9c1861cdefdf81f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:13.224946 22628 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6917978d11d493f9c1861cdefdf81f2 [term 1 LEADER]: Becoming Leader. State: Replica: e6917978d11d493f9c1861cdefdf81f2, State: Running, Role: LEADER
I20260925 19:09:13.225054 22628 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6917978d11d493f9c1861cdefdf81f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6917978d11d493f9c1861cdefdf81f2" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 43041 } }
I20260925 19:09:13.225406 22628 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6917978d11d493f9c1861cdefdf81f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6917978d11d493f9c1861cdefdf81f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6917978d11d493f9c1861cdefdf81f2" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 43041 } } }
I20260925 19:09:13.225503 22628 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6917978d11d493f9c1861cdefdf81f2 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:13.225636 22628 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6917978d11d493f9c1861cdefdf81f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6917978d11d493f9c1861cdefdf81f2. Latest consensus state: current_term: 1 leader_uuid: "e6917978d11d493f9c1861cdefdf81f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6917978d11d493f9c1861cdefdf81f2" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 43041 } } }
I20260925 19:09:13.225720 22628 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6917978d11d493f9c1861cdefdf81f2 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:13.226516 22639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:13.227293 22630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:13.224648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51711 (local address 127.0.196.254:43041)
0925 19:09:13.224868 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:13.224880 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:13.224917 (+ 37us) server_negotiation.cc:407] Connection header received
0925 19:09:13.225049 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:13.225057 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:13.225100 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:13.225261 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:13.226818 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:13.226821 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:13.226821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:13.226825 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:13.226855 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:13.226890 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:13.226981 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:13.227059 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 19:09:13.227102 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260925 19:09:13.227399 22639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:13.229496 787 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:09:13.229512 22639 catalog_manager.cc:1347] Generated new cluster ID: ad0a04f181434f95803a4392c9dafd30
I20260925 19:09:13.229576 22639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:09:13.232322 22630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:13.231156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48805 (local address 127.0.196.254:43041)
0925 19:09:13.231192 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:13.231194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:13.231332 (+ 138us) server_negotiation.cc:407] Connection header received
0925 19:09:13.231398 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:13.231399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:13.231420 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:13.231474 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:13.232091 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:13.232093 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:13.232093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:13.232098 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:13.232113 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:13.232117 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:13.232171 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:13.232244 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 19:09:13.232258 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:09:13.242861 22639 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:13.243075 22639 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:13.252092 22639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6917978d11d493f9c1861cdefdf81f2: Generated new TSK 0
I20260925 19:09:13.252377 22639 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:09:13.263384 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 22557
2026-09-25T19:09:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:09:13 dist-test-slave-rn98 krb5kdc[22656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:09:13 dist-test-slave-rn98 krb5kdc[22656](info): set up 2 sockets
Sep 25 19:09:13 dist-test-slave-rn98 krb5kdc[22656](info): commencing operation
krb5kdc: starting...
W20260925 19:09:14.323220 787 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:09:14 dist-test-slave-rn98 krb5kdc[22656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:09:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:09:14.446130 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:45673
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:41337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:45673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:14.530218 22669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:14.530441 22669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:14.530467 22669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:14.531714 22669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:14.531771 22669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:14.531786 22669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:14.531800 22669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:14.531812 22669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:14.533356 22669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:41337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:45673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:45673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:14.533634 22669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:14.533882 22669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:14.539080 22674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:14.539120 22675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:14.539285 22669 server_base.cc:1034] running on GCE node
W20260925 19:09:14.539124 22677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:14.539646 22669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:14.539919 22669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:14.541076 22669 hybrid_clock.cc:648] HybridClock initialized: now 1790363354541062 us; error 28 us; skew 500 ppm
Sep 25 19:09:14 dist-test-slave-rn98 krb5kdc[22656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:14.543715 22669 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:09:14.544301 22669 webserver.cc:460] Webserver started at http://127.0.196.254:38999/ using document root <none> and password file <none>
I20260925 19:09:14.544544 22669 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:14.544618 22669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:14.544775 22669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:14.545917 22669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "a0feebbf43dc46a7a3f669a4fd2f42d0"
format_stamp: "Formatted at 2026-09-25 19:09:14 on dist-test-slave-rn98"
server_key: "55212c1c1f8a3b43f670640aaa9f14b2"
server_key_iv: "2ff40917ef343465af8b34d5782882a6"
server_key_version: "encryptionkey@0"
I20260925 19:09:14.546308 22669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "a0feebbf43dc46a7a3f669a4fd2f42d0"
format_stamp: "Formatted at 2026-09-25 19:09:14 on dist-test-slave-rn98"
server_key: "55212c1c1f8a3b43f670640aaa9f14b2"
server_key_iv: "2ff40917ef343465af8b34d5782882a6"
server_key_version: "encryptionkey@0"
I20260925 19:09:14.547782 22669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:09:14.548636 22684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:14.548812 22669 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:14.548969 22669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "a0feebbf43dc46a7a3f669a4fd2f42d0"
format_stamp: "Formatted at 2026-09-25 19:09:14 on dist-test-slave-rn98"
server_key: "55212c1c1f8a3b43f670640aaa9f14b2"
server_key_iv: "2ff40917ef343465af8b34d5782882a6"
server_key_version: "encryptionkey@0"
I20260925 19:09:14.549065 22669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:14.561997 22669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:14.564414 22669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:14.564589 22669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:14.568924 22669 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:45673
I20260925 19:09:14.568964 22736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:45673 every 8 connection(s)
I20260925 19:09:14.569341 22669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:14.570302 22737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:14.571254 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 22669
I20260925 19:09:14.571419 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:09:14.571628 787 external_mini_cluster.cc:1428] Setting key 7f0b063635a01169dc5a4e2080b53e98
I20260925 19:09:14.573220 22737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0feebbf43dc46a7a3f669a4fd2f42d0: Bootstrap starting.
I20260925 19:09:14.574054 22737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0feebbf43dc46a7a3f669a4fd2f42d0: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:14.574384 22737 log.cc:824] T 00000000000000000000000000000000 P a0feebbf43dc46a7a3f669a4fd2f42d0: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:14.575150 22737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0feebbf43dc46a7a3f669a4fd2f42d0: No bootstrap required, opened a new log
I20260925 19:09:14.577108 22737 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0feebbf43dc46a7a3f669a4fd2f42d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0feebbf43dc46a7a3f669a4fd2f42d0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 45673 } }
I20260925 19:09:14.577311 22737 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0feebbf43dc46a7a3f669a4fd2f42d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:14.577368 22737 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0feebbf43dc46a7a3f669a4fd2f42d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0feebbf43dc46a7a3f669a4fd2f42d0, State: Initialized, Role: FOLLOWER
I20260925 19:09:14.577526 22737 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0feebbf43dc46a7a3f669a4fd2f42d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0feebbf43dc46a7a3f669a4fd2f42d0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 45673 } }
I20260925 19:09:14.577616 22737 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0feebbf43dc46a7a3f669a4fd2f42d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:14.577661 22737 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0feebbf43dc46a7a3f669a4fd2f42d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:14.577709 22737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0feebbf43dc46a7a3f669a4fd2f42d0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:14.578306 22737 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0feebbf43dc46a7a3f669a4fd2f42d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0feebbf43dc46a7a3f669a4fd2f42d0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 45673 } }
I20260925 19:09:14.578435 22737 leader_election.cc:302] T 00000000000000000000000000000000 P a0feebbf43dc46a7a3f669a4fd2f42d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0feebbf43dc46a7a3f669a4fd2f42d0; no voters:
I20260925 19:09:14.578632 22737 leader_election.cc:288] T 00000000000000000000000000000000 P a0feebbf43dc46a7a3f669a4fd2f42d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:14.578826 22742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0feebbf43dc46a7a3f669a4fd2f42d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:14.578912 22737 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0feebbf43dc46a7a3f669a4fd2f42d0 [sys.catalog]: configured and running, proceeding with master startup.
Sep 25 19:09:14 dist-test-slave-rn98 krb5kdc[22656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:09:14.579072 22742 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0feebbf43dc46a7a3f669a4fd2f42d0 [term 1 LEADER]: Becoming Leader. State: Replica: a0feebbf43dc46a7a3f669a4fd2f42d0, State: Running, Role: LEADER
I20260925 19:09:14.579169 22742 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0feebbf43dc46a7a3f669a4fd2f42d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0feebbf43dc46a7a3f669a4fd2f42d0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 45673 } }
I20260925 19:09:14.579487 22744 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0feebbf43dc46a7a3f669a4fd2f42d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0feebbf43dc46a7a3f669a4fd2f42d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0feebbf43dc46a7a3f669a4fd2f42d0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 45673 } } }
I20260925 19:09:14.579475 22742 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0feebbf43dc46a7a3f669a4fd2f42d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0feebbf43dc46a7a3f669a4fd2f42d0. Latest consensus state: current_term: 1 leader_uuid: "a0feebbf43dc46a7a3f669a4fd2f42d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0feebbf43dc46a7a3f669a4fd2f42d0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 45673 } } }
I20260925 19:09:14.579566 22744 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0feebbf43dc46a7a3f669a4fd2f42d0 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:14.579573 22742 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0feebbf43dc46a7a3f669a4fd2f42d0 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:14.579882 22751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:14.580704 22751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:14.583235 22751 catalog_manager.cc:1347] Generated new cluster ID: d7444916f5224579b89ebda45e3d59e7
I20260925 19:09:14.583338 22751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:09:14.583644 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:14.572252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37323 (local address 127.0.196.254:45673)
0925 19:09:14.572445 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:14.572461 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:14.572488 (+ 27us) server_negotiation.cc:407] Connection header received
0925 19:09:14.572980 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:14.572987 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:14.573040 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:14.573173 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:14.574029 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:14.575217 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:14.576039 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:14.576467 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:14.576586 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:14.579773 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:14.579777 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:14.579777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:14.579792 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:14.581873 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:14.582322 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:14.582325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:14.582326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:14.582447 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:14.582686 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:14.582687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:14.582688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:14.583032 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:14.583174 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:14.583318 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 19:09:14.583480 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260925 19:09:14.584734 787 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:09:14.588495 22751 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:14.589201 22751 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:14.593237 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:14.586123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35501 (local address 127.0.196.254:45673)
0925 19:09:14.586163 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:14.586165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:14.586293 (+ 128us) server_negotiation.cc:407] Connection header received
0925 19:09:14.586378 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:14.586379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:14.586389 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:14.586437 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:14.587090 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:14.587868 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:14.588882 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:14.589115 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:14.589201 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:14.590491 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:14.590492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:14.590493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:14.590500 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:14.592002 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:14.592385 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:14.592386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:14.592386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:14.592451 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:14.592652 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:14.592652 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:14.592653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:14.592772 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:14.592886 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:14.593024 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 19:09:14.593121 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:09:14.594110 22751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0feebbf43dc46a7a3f669a4fd2f42d0: Generated new TSK 0
I20260925 19:09:14.594306 22751 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:09:14.599233 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 22669
2026-09-25T19:09:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:09:14 dist-test-slave-rn98 krb5kdc[22768](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:09:14 dist-test-slave-rn98 krb5kdc[22768](info): set up 2 sockets
Sep 25 19:09:14 dist-test-slave-rn98 krb5kdc[22768](info): commencing operation
krb5kdc: starting...
W20260925 19:09:15.638943 787 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:09:15 dist-test-slave-rn98 krb5kdc[22768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:09:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:09:15.762415 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:37655
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:37773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:37655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:15.846095 22781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:15.846509 22781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:15.846607 22781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:15.848011 22781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:15.848093 22781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:15.848121 22781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:15.848155 22781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:15.848178 22781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:15.849921 22781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:37773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:37655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:37655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:15.850251 22781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:15.850508 22781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:15.856061 22787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:15.856061 22786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:15.856061 22789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:15.856096 22781 server_base.cc:1034] running on GCE node
I20260925 19:09:15.856626 22781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:15.856942 22781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:15.858112 22781 hybrid_clock.cc:648] HybridClock initialized: now 1790363355858113 us; error 53 us; skew 500 ppm
Sep 25 19:09:15 dist-test-slave-rn98 krb5kdc[22768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:15.860679 22781 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:09:15.861220 22781 webserver.cc:460] Webserver started at http://127.0.196.254:39049/ using document root <none> and password file <none>
I20260925 19:09:15.861464 22781 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:15.861531 22781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:15.861675 22781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:15.862855 22781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "9a0a19f590714b82b571e3b4d94809f0"
format_stamp: "Formatted at 2026-09-25 19:09:15 on dist-test-slave-rn98"
server_key: "38a32bce1301d85e5a52f79ef30f86b7"
server_key_iv: "cbe32a8230312c52c810eaa1ab7a1ad0"
server_key_version: "encryptionkey@0"
I20260925 19:09:15.863230 22781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "9a0a19f590714b82b571e3b4d94809f0"
format_stamp: "Formatted at 2026-09-25 19:09:15 on dist-test-slave-rn98"
server_key: "38a32bce1301d85e5a52f79ef30f86b7"
server_key_iv: "cbe32a8230312c52c810eaa1ab7a1ad0"
server_key_version: "encryptionkey@0"
I20260925 19:09:15.864656 22781 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:15.865393 22796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:15.865571 22781 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:09:15.865667 22781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "9a0a19f590714b82b571e3b4d94809f0"
format_stamp: "Formatted at 2026-09-25 19:09:15 on dist-test-slave-rn98"
server_key: "38a32bce1301d85e5a52f79ef30f86b7"
server_key_iv: "cbe32a8230312c52c810eaa1ab7a1ad0"
server_key_version: "encryptionkey@0"
I20260925 19:09:15.865752 22781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:15.881798 22781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:15.883920 22781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:15.884097 22781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:15.888700 22781 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:37655
I20260925 19:09:15.888756 22848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:37655 every 8 connection(s)
I20260925 19:09:15.889151 22781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:15.890134 22849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:15.892524 22849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a0a19f590714b82b571e3b4d94809f0: Bootstrap starting.
I20260925 19:09:15.893131 22849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a0a19f590714b82b571e3b4d94809f0: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:15.893424 22849 log.cc:824] T 00000000000000000000000000000000 P 9a0a19f590714b82b571e3b4d94809f0: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:15.894193 22849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a0a19f590714b82b571e3b4d94809f0: No bootstrap required, opened a new log
I20260925 19:09:15.895931 22849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a0a19f590714b82b571e3b4d94809f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0a19f590714b82b571e3b4d94809f0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37655 } }
I20260925 19:09:15.896085 22849 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a0a19f590714b82b571e3b4d94809f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:15.896108 22849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a0a19f590714b82b571e3b4d94809f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a0a19f590714b82b571e3b4d94809f0, State: Initialized, Role: FOLLOWER
I20260925 19:09:15.896288 22849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a0a19f590714b82b571e3b4d94809f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0a19f590714b82b571e3b4d94809f0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37655 } }
I20260925 19:09:15.896358 22849 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a0a19f590714b82b571e3b4d94809f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:15.896379 22849 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a0a19f590714b82b571e3b4d94809f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:15.896432 22849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a0a19f590714b82b571e3b4d94809f0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:15.897046 22849 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a0a19f590714b82b571e3b4d94809f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0a19f590714b82b571e3b4d94809f0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37655 } }
I20260925 19:09:15.897142 22849 leader_election.cc:302] T 00000000000000000000000000000000 P 9a0a19f590714b82b571e3b4d94809f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a0a19f590714b82b571e3b4d94809f0; no voters:
I20260925 19:09:15.897347 22849 leader_election.cc:288] T 00000000000000000000000000000000 P 9a0a19f590714b82b571e3b4d94809f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:15.897445 22852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a0a19f590714b82b571e3b4d94809f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:15.897660 22852 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a0a19f590714b82b571e3b4d94809f0 [term 1 LEADER]: Becoming Leader. State: Replica: 9a0a19f590714b82b571e3b4d94809f0, State: Running, Role: LEADER
I20260925 19:09:15.897715 22849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a0a19f590714b82b571e3b4d94809f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:15.897890 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 22781
I20260925 19:09:15.897797 22852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a0a19f590714b82b571e3b4d94809f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0a19f590714b82b571e3b4d94809f0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37655 } }
I20260925 19:09:15.898015 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:09:15.898236 787 external_mini_cluster.cc:1428] Setting key 128901e4392bf2747078ddb4d925ac9d
I20260925 19:09:15.898542 22854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a0a19f590714b82b571e3b4d94809f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a0a19f590714b82b571e3b4d94809f0. Latest consensus state: current_term: 1 leader_uuid: "9a0a19f590714b82b571e3b4d94809f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0a19f590714b82b571e3b4d94809f0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37655 } } }
I20260925 19:09:15.898655 22854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a0a19f590714b82b571e3b4d94809f0 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:15.899008 22861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:15.899047 22853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a0a19f590714b82b571e3b4d94809f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a0a19f590714b82b571e3b4d94809f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0a19f590714b82b571e3b4d94809f0" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37655 } } }
I20260925 19:09:15.899664 22853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a0a19f590714b82b571e3b4d94809f0 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:15.899518 22861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:15.901569 22861 catalog_manager.cc:1347] Generated new cluster ID: 5c142e0e7edf4f4eb240039354f15cb6
I20260925 19:09:15.901626 22861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:09:15.904618 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:15.898804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43113 (local address 127.0.196.254:37655)
0925 19:09:15.899623 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:15.899633 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:15.899668 (+ 35us) server_negotiation.cc:407] Connection header received
0925 19:09:15.899936 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:15.899945 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:15.900010 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:15.900166 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:15.900409 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:15.901637 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:15.902499 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:15.903072 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:15.903197 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:15.903947 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:15.903951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:15.903951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:15.904134 (+ 183us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:15.904166 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:15.904177 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:15.904251 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:15.904331 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 19:09:15.904400 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":765,"thread_start_us":75,"threads_started":1}
I20260925 19:09:15.905507 787 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:09:15.909790 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:15.906877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55651 (local address 127.0.196.254:37655)
0925 19:09:15.906915 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:15.906916 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:15.907034 (+ 118us) server_negotiation.cc:407] Connection header received
0925 19:09:15.907090 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:15.907091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:15.907102 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:15.907157 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:15.907273 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:15.907764 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:15.908477 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:15.908699 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:15.908774 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:15.909510 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:15.909511 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:15.909511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:15.909516 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:15.909530 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:15.909534 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:15.909588 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:15.909648 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 19:09:15.909671 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:09:15.911656 22861 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:15.912335 22861 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:15.918721 22861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a0a19f590714b82b571e3b4d94809f0: Generated new TSK 0
I20260925 19:09:15.918967 22861 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:09:15.921411 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 22781
2026-09-25T19:09:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:09:15 dist-test-slave-rn98 krb5kdc[22880](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:09:15 dist-test-slave-rn98 krb5kdc[22880](info): set up 2 sockets
Sep 25 19:09:15 dist-test-slave-rn98 krb5kdc[22880](info): commencing operation
krb5kdc: starting...
W20260925 19:09:17.007004 787 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:09:17 dist-test-slave-rn98 krb5kdc[22880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:09:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:09:17.135818 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:39887
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:46187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:39887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:17.219328 22893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:17.219558 22893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:17.219583 22893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:17.220821 22893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:17.220873 22893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:17.220888 22893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:17.220902 22893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:17.220914 22893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:17.222537 22893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:46187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:39887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:39887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:17.222848 22893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:17.223093 22893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:17.228348 22898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:17.228343 22899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:17.228397 22901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:17.229012 22893 server_base.cc:1034] running on GCE node
I20260925 19:09:17.229265 22893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:17.229559 22893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:17.230743 22893 hybrid_clock.cc:648] HybridClock initialized: now 1790363357230710 us; error 42 us; skew 500 ppm
Sep 25 19:09:17 dist-test-slave-rn98 krb5kdc[22880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:17.237805 22893 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:09:17.238372 22893 webserver.cc:460] Webserver started at http://127.0.196.254:39991/ using document root <none> and password file <none>
I20260925 19:09:17.238685 22893 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:17.238763 22893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:17.238935 22893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:17.240106 22893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "798956feface424796991dc9383a031f"
format_stamp: "Formatted at 2026-09-25 19:09:17 on dist-test-slave-rn98"
server_key: "a0fcf56668e3422cfa1f305e45f4ce22"
server_key_iv: "9fd4357f3808fc533e0bc033394a99a0"
server_key_version: "encryptionkey@0"
I20260925 19:09:17.240520 22893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "798956feface424796991dc9383a031f"
format_stamp: "Formatted at 2026-09-25 19:09:17 on dist-test-slave-rn98"
server_key: "a0fcf56668e3422cfa1f305e45f4ce22"
server_key_iv: "9fd4357f3808fc533e0bc033394a99a0"
server_key_version: "encryptionkey@0"
I20260925 19:09:17.242092 22893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:09:17.243120 22908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:17.243352 22893 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:09:17.243477 22893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "798956feface424796991dc9383a031f"
format_stamp: "Formatted at 2026-09-25 19:09:17 on dist-test-slave-rn98"
server_key: "a0fcf56668e3422cfa1f305e45f4ce22"
server_key_iv: "9fd4357f3808fc533e0bc033394a99a0"
server_key_version: "encryptionkey@0"
I20260925 19:09:17.243561 22893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:17.247316 22893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:17.249428 22893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:17.249599 22893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:17.253908 22893 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:39887
I20260925 19:09:17.253973 22960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:39887 every 8 connection(s)
I20260925 19:09:17.254331 22893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:17.255447 22961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:17.257825 22961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 798956feface424796991dc9383a031f: Bootstrap starting.
I20260925 19:09:17.258379 22961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 798956feface424796991dc9383a031f: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:17.258672 22961 log.cc:824] T 00000000000000000000000000000000 P 798956feface424796991dc9383a031f: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:17.259413 22961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 798956feface424796991dc9383a031f: No bootstrap required, opened a new log
I20260925 19:09:17.261101 22961 raft_consensus.cc:348] T 00000000000000000000000000000000 P 798956feface424796991dc9383a031f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798956feface424796991dc9383a031f" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 39887 } }
I20260925 19:09:17.261255 22961 raft_consensus.cc:374] T 00000000000000000000000000000000 P 798956feface424796991dc9383a031f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:09:17.261312 22961 raft_consensus.cc:729] T 00000000000000000000000000000000 P 798956feface424796991dc9383a031f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 798956feface424796991dc9383a031f, State: Initialized, Role: FOLLOWER
I20260925 19:09:17.261478 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 22893
I20260925 19:09:17.261472 22961 consensus_queue.cc:260] T 00000000000000000000000000000000 P 798956feface424796991dc9383a031f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798956feface424796991dc9383a031f" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 39887 } }
I20260925 19:09:17.261560 22961 raft_consensus.cc:388] T 00000000000000000000000000000000 P 798956feface424796991dc9383a031f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:09:17.261586 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:09:17.261601 22961 raft_consensus.cc:482] T 00000000000000000000000000000000 P 798956feface424796991dc9383a031f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:09:17.261647 22961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 798956feface424796991dc9383a031f [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:17.261821 787 external_mini_cluster.cc:1428] Setting key 8ad6df4c42c96806d0351a746fdee408
I20260925 19:09:17.262476 22961 raft_consensus.cc:504] T 00000000000000000000000000000000 P 798956feface424796991dc9383a031f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798956feface424796991dc9383a031f" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 39887 } }
I20260925 19:09:17.262643 22961 leader_election.cc:302] T 00000000000000000000000000000000 P 798956feface424796991dc9383a031f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 798956feface424796991dc9383a031f; no voters:
I20260925 19:09:17.262818 22961 leader_election.cc:288] T 00000000000000000000000000000000 P 798956feface424796991dc9383a031f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:09:17.262867 22966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 798956feface424796991dc9383a031f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:17.263026 22966 raft_consensus.cc:686] T 00000000000000000000000000000000 P 798956feface424796991dc9383a031f [term 1 LEADER]: Becoming Leader. State: Replica: 798956feface424796991dc9383a031f, State: Running, Role: LEADER
I20260925 19:09:17.263089 22961 sys_catalog.cc:564] T 00000000000000000000000000000000 P 798956feface424796991dc9383a031f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:17.263123 22966 consensus_queue.cc:237] T 00000000000000000000000000000000 P 798956feface424796991dc9383a031f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798956feface424796991dc9383a031f" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 39887 } }
I20260925 19:09:17.263490 22966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798956feface424796991dc9383a031f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 798956feface424796991dc9383a031f. Latest consensus state: current_term: 1 leader_uuid: "798956feface424796991dc9383a031f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798956feface424796991dc9383a031f" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 39887 } } }
I20260925 19:09:17.263581 22966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798956feface424796991dc9383a031f [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:17.263767 22967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798956feface424796991dc9383a031f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "798956feface424796991dc9383a031f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798956feface424796991dc9383a031f" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 39887 } } }
I20260925 19:09:17.263803 22972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:17.264513 22972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:17.264731 22965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:17.262344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38189 (local address 127.0.196.254:39887)
0925 19:09:17.262598 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:17.262610 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:17.262678 (+ 68us) server_negotiation.cc:407] Connection header received
0925 19:09:17.262813 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:17.262820 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:17.262855 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:17.263008 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:17.264099 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:17.264104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:17.264105 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:17.264110 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:17.264162 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:17.264187 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:17.264455 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:17.264499 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 19:09:17.264545 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":131,"threads_started":1}
I20260925 19:09:17.265383 22967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798956feface424796991dc9383a031f [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:17.265785 787 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:09:17.267119 22972 catalog_manager.cc:1347] Generated new cluster ID: eed056597b0744d4a53474804f8351fe
I20260925 19:09:17.267176 22972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:09:17.268689 22965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:17.267374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59583 (local address 127.0.196.254:39887)
0925 19:09:17.267411 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:17.267413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:17.267557 (+ 144us) server_negotiation.cc:407] Connection header received
0925 19:09:17.267591 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:17.267592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:17.267597 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:17.267665 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:17.268421 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:17.268422 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:17.268423 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:09:17.268427 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:17.268443 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:09:17.268450 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:17.268480 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:17.268602 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 19:09:17.268619 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:09:17.277407 22972 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:17.277632 22972 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:17.280893 22972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 798956feface424796991dc9383a031f: Generated new TSK 0
I20260925 19:09:17.281109 22972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:09:17.284143 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 22893
2026-09-25T19:09:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1362 ms)
[----------] 18 tests from AuthTokenIssuingTest (24568 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:09:17 dist-test-slave-rn98 krb5kdc[22992](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:09:17 dist-test-slave-rn98 krb5kdc[22992](info): set up 2 sockets
Sep 25 19:09:17 dist-test-slave-rn98 krb5kdc[22992](info): commencing operation
krb5kdc: starting...
W20260925 19:09:18.334959 787 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:09:18 dist-test-slave-rn98 krb5kdc[22992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:09:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:09:18.464016 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:37985
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:46503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:37985,127.0.196.253:35505,127.0.196.252:34119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:18.553793 23005 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 19:09:18.554049 23005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:18.554067 23005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:18.554090 23005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:18.555400 23005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:18.555454 23005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:18.555469 23005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:18.555482 23005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:18.555495 23005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:18.557067 23005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:46503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:37985,127.0.196.253:35505,127.0.196.252:34119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:37985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:18.557286 23005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:18.557478 23005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:18.562935 23010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:18.562928 23013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:18.562954 23005 server_base.cc:1034] running on GCE node
W20260925 19:09:18.562923 23011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:18.563553 23005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:18.563854 23005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:18.565037 23005 hybrid_clock.cc:648] HybridClock initialized: now 1790363358565016 us; error 38 us; skew 500 ppm
Sep 25 19:09:18 dist-test-slave-rn98 krb5kdc[22992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:18.568001 23005 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:09:18.568496 23005 webserver.cc:460] Webserver started at http://127.0.196.254:33811/ using document root <none> and password file <none>
I20260925 19:09:18.568742 23005 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:18.568814 23005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:18.568982 23005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:18.570124 23005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "b8d9e0153112454c8dbc2e7e74333e13"
format_stamp: "Formatted at 2026-09-25 19:09:18 on dist-test-slave-rn98"
server_key: "badf5b7b7d6df3f1b67abd76f5632637"
server_key_iv: "b2b04766d5ce08177d9e586ec483c75f"
server_key_version: "encryptionkey@0"
I20260925 19:09:18.570519 23005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "b8d9e0153112454c8dbc2e7e74333e13"
format_stamp: "Formatted at 2026-09-25 19:09:18 on dist-test-slave-rn98"
server_key: "badf5b7b7d6df3f1b67abd76f5632637"
server_key_iv: "b2b04766d5ce08177d9e586ec483c75f"
server_key_version: "encryptionkey@0"
I20260925 19:09:18.571993 23005 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:09:18.573079 23020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:18.573323 23005 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:09:18.573443 23005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "b8d9e0153112454c8dbc2e7e74333e13"
format_stamp: "Formatted at 2026-09-25 19:09:18 on dist-test-slave-rn98"
server_key: "badf5b7b7d6df3f1b67abd76f5632637"
server_key_iv: "b2b04766d5ce08177d9e586ec483c75f"
server_key_version: "encryptionkey@0"
I20260925 19:09:18.573527 23005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:18.591686 23005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:18.594074 23005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:18.594257 23005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:18.598587 23005 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:37985
I20260925 19:09:18.598673 23072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:37985 every 8 connection(s)
I20260925 19:09:18.599088 23005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:18.599601 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 23005
I20260925 19:09:18.599730 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:09:18.599987 787 external_mini_cluster.cc:1428] Setting key 90f571515747d9db9c50975cdf490c1d
I20260925 19:09:18.600544 23073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:18.602869 23073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37985 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.196.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.253@KRBTEST.COM" created.
Sep 25 19:09:18 dist-test-slave-rn98 krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:09:18.615404 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.603687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:37985 (local address 127.0.0.1:48282)
0925 19:09:18.603898 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:18.603951 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:18.604510 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:18.604847 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:18.604850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:18.605162 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:18.605358 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:18.605364 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.606518 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.606519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:18.607388 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:18.607390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.608019 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.608020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:18.608092 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:18.608896 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:18.608906 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:18.611337 (+ 2431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:18.613747 (+ 2410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:18.613754 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:18.613757 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:18.613968 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:18.614267 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:18.614268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:18.614268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:18.614359 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:18.614910 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:18.614915 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:18.614981 (+ 66us) client_negotiation.cc:769] Sending connection context
0925 19:09:18.615072 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 19:09:18.615219 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":93,"threads_started":1}
I20260925 19:09:18.615521 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.603813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48282 (local address 127.0.196.254:37985)
0925 19:09:18.604002 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:18.604008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:18.604048 (+ 40us) server_negotiation.cc:407] Connection header received
0925 19:09:18.604705 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:18.604708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:18.604727 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:18.604842 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:18.605476 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.606387 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.607532 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.607888 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.608040 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:18.611458 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:18.611469 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:18.611470 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:18.611480 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:18.613635 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:18.614060 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:18.614064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:18.614064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:18.614191 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:18.614434 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:18.614435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:18.614436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:18.614820 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:18.614918 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:18.615135 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 19:09:18.615312 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":159,"thread_start_us":84,"threads_started":1}
I20260925 19:09:18.616460 23073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.253" port: 35505 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:09:18.616846 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.616727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:35505 (local address 127.0.0.1:42628)
0925 19:09:18.616777 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:18.616823 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.253:35505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 19:09:18.617015 23023 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.196.253:35505: connect: Connection refused (error 111)
I20260925 19:09:18.617520 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.617416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:35505 (local address 127.0.0.1:42640)
0925 19:09:18.617466 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:18.617499 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.253:35505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 19:09:18.617672 23073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.196.253:35505: Network error: Client connection negotiation failed: client connection to 127.0.196.253:35505: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/krb5kdc/kudu_127.0.196.253.keytab.
Entry for principal kudu/127.0.196.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/krb5kdc/kudu_127.0.196.253.keytab.
I20260925 19:09:18.629431 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.253:35505
--webserver_interface=127.0.196.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/krb5kdc/kudu_127.0.196.253.keytab
--principal=kudu/127.0.196.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:46503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:37985,127.0.196.253:35505,127.0.196.252:34119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:09:18.666934 23073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.253" port: 35505 } attempt: 1
I20260925 19:09:18.667538 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.667374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:35505 (local address 127.0.0.1:42646)
0925 19:09:18.667457 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:18.667503 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.253:35505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 19:09:18.668042 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.667888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:35505 (local address 127.0.0.1:42648)
0925 19:09:18.667968 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:18.668002 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.253:35505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 19:09:18.668164 23073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.196.253:35505: Network error: Client connection negotiation failed: client connection to 127.0.196.253:35505: connect: Connection refused (error 111)
W20260925 19:09:18.715999 23080 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 19:09:18.716288 23080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:18.716321 23080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:18.716368 23080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:18.717566 23080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:18.717644 23080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:18.717677 23080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:18.717706 23080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:18.717737 23080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:18.719396 23080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:46503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:37985,127.0.196.253:35505,127.0.196.252:34119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/krb5kdc/kudu_127.0.196.253.keytab
--principal=kudu/127.0.196.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.253:35505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.196.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:18.719703 23080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:18.719959 23080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:18.725171 23088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:18.725132 23085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:18.725360 23080 server_base.cc:1034] running on GCE node
W20260925 19:09:18.725131 23086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:18.725775 23080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:18.726126 23080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:18.727319 23080 hybrid_clock.cc:648] HybridClock initialized: now 1790363358727289 us; error 50 us; skew 500 ppm
Sep 25 19:09:18 dist-test-slave-rn98 krb5kdc[22992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:18.730053 23080 init.cc:377] Logged in from keytab as kudu/127.0.196.253@KRBTEST.COM (short username kudu)
I20260925 19:09:18.730522 23080 webserver.cc:460] Webserver started at http://127.0.196.253:42677/ using document root <none> and password file <none>
I20260925 19:09:18.730803 23080 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:18.730875 23080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:18.731036 23080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:18.732170 23080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-1/data/instance:
uuid: "f51885b9308e4f1ca58ec53f93827dae"
format_stamp: "Formatted at 2026-09-25 19:09:18 on dist-test-slave-rn98"
server_key: "d06221ade4d03decdec917f377bb86f3"
server_key_iv: "3474f0ad0ffea9543fdb5a4fb2f512df"
server_key_version: "encryptionkey@0"
I20260925 19:09:18.732558 23080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-1/wal/instance:
uuid: "f51885b9308e4f1ca58ec53f93827dae"
format_stamp: "Formatted at 2026-09-25 19:09:18 on dist-test-slave-rn98"
server_key: "d06221ade4d03decdec917f377bb86f3"
server_key_iv: "3474f0ad0ffea9543fdb5a4fb2f512df"
server_key_version: "encryptionkey@0"
I20260925 19:09:18.734019 23080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:09:18.734972 23095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:18.735164 23080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:18.735270 23080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-1/wal
uuid: "f51885b9308e4f1ca58ec53f93827dae"
format_stamp: "Formatted at 2026-09-25 19:09:18 on dist-test-slave-rn98"
server_key: "d06221ade4d03decdec917f377bb86f3"
server_key_iv: "3474f0ad0ffea9543fdb5a4fb2f512df"
server_key_version: "encryptionkey@0"
I20260925 19:09:18.735354 23080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:18.736325 23073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.253" port: 35505 } attempt: 2
I20260925 19:09:18.736740 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.736602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:35505 (local address 127.0.0.1:42664)
0925 19:09:18.736661 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:18.736705 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.253:35505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 19:09:18.737049 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.736992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:35505 (local address 127.0.0.1:42680)
0925 19:09:18.737013 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:18.737035 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.253:35505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260925 19:09:18.737154 23073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.196.253:35505: Network error: Client connection negotiation failed: client connection to 127.0.196.253:35505: connect: Connection refused (error 111)
I20260925 19:09:18.746810 23080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:18.748883 23080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:18.749042 23080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:18.753444 23080 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.253:35505
I20260925 19:09:18.753525 23147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.253:35505 every 8 connection(s)
I20260925 19:09:18.753892 23080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-1/data/info.pb
I20260925 19:09:18.754734 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 23080
I20260925 19:09:18.754837 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-1/wal/instance
I20260925 19:09:18.755039 787 external_mini_cluster.cc:1428] Setting key fa480b87cefa17c6f4e33dd95d91acd9
I20260925 19:09:18.755147 23148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:18.757280 23148 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37985 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.196.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.252@KRBTEST.COM" created.
Sep 25 19:09:18 dist-test-slave-rn98 krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.253@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:09:18.769676 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.758048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48290 (local address 127.0.196.254:37985)
0925 19:09:18.758121 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:18.758123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:18.758354 (+ 231us) server_negotiation.cc:407] Connection header received
0925 19:09:18.758968 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:18.758969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:18.758987 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:18.759038 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:18.759691 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.760198 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.761190 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.761411 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.761501 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:18.765974 (+ 4473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:18.765976 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:18.765977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:18.765988 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:18.767807 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:18.768406 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:18.768407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:18.768408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:18.768482 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:18.768856 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:18.768858 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:18.768858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:18.769008 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:18.769081 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:18.769506 (+ 425us) server_negotiation.cc:299] Negotiation successful
0925 19:09:18.769592 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 19:09:18.769949 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.758030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:37985 (local address 127.0.0.1:48290)
0925 19:09:18.758219 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:18.758264 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:18.758849 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:18.759118 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:18.759121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:18.759389 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:18.759593 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:18.759598 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.760302 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.760303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:18.761097 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:18.761100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.761532 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.761534 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:18.761654 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:18.762242 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:18.762261 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:18.765854 (+ 3593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:18.767915 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:18.767925 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:18.767928 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:18.768293 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:18.768563 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:18.768564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:18.768565 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:18.768738 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:18.769095 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:18.769103 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:18.769328 (+ 225us) client_negotiation.cc:769] Sending connection context
0925 19:09:18.769465 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 19:09:18.769679 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260925 19:09:18.770684 23148 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.253" port: 35505 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 19:09:18 dist-test-slave-rn98 krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.253@KRBTEST.COM for kudu/127.0.196.253@KRBTEST.COM
I20260925 19:09:18.780238 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.771007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:35505 (local address 127.0.0.1:42696)
0925 19:09:18.771068 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:18.771081 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:18.771130 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:18.771375 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:18.771376 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:18.771537 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:18.771609 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:18.771612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.772562 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.772563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:18.773429 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:18.773433 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.773959 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.773961 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:18.774039 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:18.774445 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:18.774449 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:18.777108 (+ 2659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:18.779173 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:18.779174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:18.779175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:18.779357 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:18.779565 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:18.779566 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:18.779566 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:18.779633 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:18.779924 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:18.779925 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:18.779964 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 19:09:18.780016 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 19:09:18.780137 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 19:09:18.780431 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.771046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42696 (local address 127.0.196.253:35505)
0925 19:09:18.771227 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:18.771232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:18.771251 (+ 19us) server_negotiation.cc:407] Connection header received
0925 19:09:18.771279 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:18.771280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:18.771291 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:18.771352 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:18.771710 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.772463 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.773517 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.773871 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.773933 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:18.777199 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:18.777209 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:18.777209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:18.777217 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:18.779091 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:18.779440 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:18.779443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:18.779443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:18.779497 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:18.779701 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:18.779702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:18.779703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:18.779861 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:18.779929 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:18.780040 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 19:09:18.780247 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":148,"thread_start_us":107,"threads_started":1}
I20260925 19:09:18.781102 23148 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.252" port: 34119 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:09:18.781525 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.781393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:34119 (local address 127.0.0.1:34464)
0925 19:09:18.781446 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:18.781501 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:34119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/krb5kdc/kudu_127.0.196.252.keytab.
Entry for principal kudu/127.0.196.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/krb5kdc/kudu_127.0.196.252.keytab.
W20260925 19:09:18.781689 23096 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.196.252:34119: connect: Connection refused (error 111)
I20260925 19:09:18.782167 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.782102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:34119 (local address 127.0.0.1:34474)
0925 19:09:18.782128 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:18.782151 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:34119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260925 19:09:18.782291 23148 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.196.252:34119: Network error: Client connection negotiation failed: client connection to 127.0.196.252:34119: connect: Connection refused (error 111)
I20260925 19:09:18.782370 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.252:34119
--webserver_interface=127.0.196.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/krb5kdc/kudu_127.0.196.252.keytab
--principal=kudu/127.0.196.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:46503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:37985,127.0.196.253:35505,127.0.196.252:34119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:09:18.828356 23073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.253" port: 35505 } attempt: 3
I20260925 19:09:18.831542 23148 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.252" port: 34119 } attempt: 1
I20260925 19:09:18.832056 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.831911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:34119 (local address 127.0.0.1:34484)
0925 19:09:18.831981 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:18.832020 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:34119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 19:09:18.832762 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.832658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:34119 (local address 127.0.0.1:34490)
0925 19:09:18.832706 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:18.832736 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:34119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 19:09:18.832918 23148 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.196.252:34119: Network error: Client connection negotiation failed: client connection to 127.0.196.252:34119: connect: Connection refused (error 111)
Sep 25 19:09:18 dist-test-slave-rn98 krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for kudu/127.0.196.253@KRBTEST.COM
I20260925 19:09:18.836901 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.828995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42698 (local address 127.0.196.253:35505)
0925 19:09:18.829065 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:18.829069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:18.829100 (+ 31us) server_negotiation.cc:407] Connection header received
0925 19:09:18.829128 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:18.829129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:18.829140 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:18.829185 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:18.829600 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.830241 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.831363 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.831622 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.831741 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:18.834075 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:18.834076 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:18.834077 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:18.834088 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:18.835685 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:18.836068 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:18.836069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:18.836070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:18.836149 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:18.836406 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:18.836407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:18.836407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:18.836522 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:18.836574 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:18.836764 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 19:09:18.836836 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 19:09:18.836933 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.828894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:35505 (local address 127.0.0.1:42698)
0925 19:09:18.828970 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:18.828986 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:18.829052 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:18.829211 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:18.829212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:18.829418 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:18.829496 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:18.829498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.830336 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.830337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:18.831229 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:18.831232 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.831828 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.831829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:18.831925 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:18.832447 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:18.832453 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:18.833970 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:18.835789 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:18.835790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:18.835790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:18.835988 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:18.836245 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:18.836246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:18.836247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:18.836334 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:18.836588 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:18.836589 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:18.836645 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 19:09:18.836719 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 19:09:18.836829 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 19:09:18.837585 23073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.252" port: 34119 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:09:18.838090 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.837848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:34119 (local address 127.0.0.1:34492)
0925 19:09:18.838013 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:18.838058 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:34119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260925 19:09:18.838454 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.838345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:34119 (local address 127.0.0.1:34500)
0925 19:09:18.838399 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:18.838431 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:34119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 19:09:18.838572 23073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.196.252:34119: Network error: Client connection negotiation failed: client connection to 127.0.196.252:34119: connect: Connection refused (error 111)
W20260925 19:09:18.868992 23155 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 19:09:18.869273 23155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:18.869306 23155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:18.869355 23155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 19:09:18.869791 23073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.252" port: 34119 } attempt: 1
I20260925 19:09:18.870285 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.870123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:34119 (local address 127.0.0.1:34506)
0925 19:09:18.870198 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:18.870247 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:34119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 19:09:18.870709 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.870588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:34119 (local address 127.0.0.1:34516)
0925 19:09:18.870653 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:18.870684 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:34119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 19:09:18.870829 23073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.196.252:34119: Network error: Client connection negotiation failed: client connection to 127.0.196.252:34119: connect: Connection refused (error 111)
W20260925 19:09:18.871054 23155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:18.871093 23155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:18.871106 23155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:18.871119 23155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:18.871169 23155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:18.872776 23155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:46503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:37985,127.0.196.253:35505,127.0.196.252:34119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/krb5kdc/kudu_127.0.196.252.keytab
--principal=kudu/127.0.196.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.252:34119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.196.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:18.873072 23155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:18.873317 23155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:18.878548 23160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:18.878520 23163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:18.878549 23161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:18.878980 23155 server_base.cc:1034] running on GCE node
I20260925 19:09:18.879271 23155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:18.879551 23155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:18.880754 23155 hybrid_clock.cc:648] HybridClock initialized: now 1790363358880673 us; error 57 us; skew 500 ppm
Sep 25 19:09:18 dist-test-slave-rn98 krb5kdc[22992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:18.883665 23155 init.cc:377] Logged in from keytab as kudu/127.0.196.252@KRBTEST.COM (short username kudu)
I20260925 19:09:18.884188 23155 webserver.cc:460] Webserver started at http://127.0.196.252:34753/ using document root <none> and password file <none>
I20260925 19:09:18.884457 23155 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:18.884529 23155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:18.884681 23155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:18.885859 23155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-2/data/instance:
uuid: "c65ab242345f45f69c5361db189b6f0a"
format_stamp: "Formatted at 2026-09-25 19:09:18 on dist-test-slave-rn98"
server_key: "8244bbd7f6bebb3538d4204b78efa600"
server_key_iv: "b30b44b7f6288b032305133f7c8d2753"
server_key_version: "encryptionkey@0"
I20260925 19:09:18.886264 23155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-2/wal/instance:
uuid: "c65ab242345f45f69c5361db189b6f0a"
format_stamp: "Formatted at 2026-09-25 19:09:18 on dist-test-slave-rn98"
server_key: "8244bbd7f6bebb3538d4204b78efa600"
server_key_iv: "b30b44b7f6288b032305133f7c8d2753"
server_key_version: "encryptionkey@0"
I20260925 19:09:18.887835 23155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:09:18.888605 23170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:18.888810 23155 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:09:18.888921 23155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-2/wal
uuid: "c65ab242345f45f69c5361db189b6f0a"
format_stamp: "Formatted at 2026-09-25 19:09:18 on dist-test-slave-rn98"
server_key: "8244bbd7f6bebb3538d4204b78efa600"
server_key_iv: "b30b44b7f6288b032305133f7c8d2753"
server_key_version: "encryptionkey@0"
I20260925 19:09:18.888998 23155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:18.896906 23155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:18.899395 23155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:18.899570 23155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:18.901090 23148 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.252" port: 34119 } attempt: 2
I20260925 19:09:18.901625 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.901443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:34119 (local address 127.0.0.1:34522)
0925 19:09:18.901527 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:18.901575 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:34119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 19:09:18.902024 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.901913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:34119 (local address 127.0.0.1:34530)
0925 19:09:18.901963 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:18.901988 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:34119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 19:09:18.902129 23148 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.196.252:34119: Network error: Client connection negotiation failed: client connection to 127.0.196.252:34119: connect: Connection refused (error 111)
I20260925 19:09:18.904021 23155 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.252:34119
I20260925 19:09:18.904090 23222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.252:34119 every 8 connection(s)
I20260925 19:09:18.904443 23155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-2/data/info.pb
I20260925 19:09:18.905465 23223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:18.907124 23223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37985 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:09:18.907483 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 23155
I20260925 19:09:18.907616 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790363087086945-787-0/minicluster-data/master-2/wal/instance
I20260925 19:09:18.907902 787 external_mini_cluster.cc:1428] Setting key a86e91fddc94911f12fe0a6152c58c2a
Sep 25 19:09:18 dist-test-slave-rn98 krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.252@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
Sep 25 19:09:18 dist-test-slave-rn98 krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.252@KRBTEST.COM
I20260925 19:09:18.918303 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.907970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48300 (local address 127.0.196.254:37985)
0925 19:09:18.908029 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:18.908031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:18.908300 (+ 269us) server_negotiation.cc:407] Connection header received
0925 19:09:18.909012 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:18.909014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:18.909025 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:18.909080 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:18.909964 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.910758 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.912090 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.912386 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.912470 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:18.915036 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:18.915039 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:18.915039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:18.915049 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:18.916748 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:18.917248 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:18.917250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:18.917250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:18.917319 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:18.917583 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:18.917584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:18.917585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:18.917716 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:18.917808 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:18.918112 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 19:09:18.918201 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 19:09:18.918553 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.908331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34538 (local address 127.0.196.252:34119)
0925 19:09:18.908483 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:18.908487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:18.908533 (+ 46us) server_negotiation.cc:407] Connection header received
0925 19:09:18.908921 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:18.908932 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:18.908976 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:18.909082 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:18.909943 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.911112 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.912216 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.912676 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.912729 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:18.915524 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:18.915533 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:18.915533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:18.915548 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:18.917126 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:18.917487 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:18.917491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:18.917491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:18.917551 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:18.917757 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:18.917758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:18.917759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:18.917957 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:18.918119 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:18.918190 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 19:09:18.918348 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
I20260925 19:09:18.918725 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.907917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:37985 (local address 127.0.0.1:48300)
0925 19:09:18.908125 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:18.908166 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:18.908906 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:18.909105 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:18.909108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:18.909577 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:18.909862 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:18.909871 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.910848 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.910849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:18.911933 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:18.911936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.912498 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.912499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:18.912602 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:18.913157 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:18.913169 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:18.914929 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:18.916863 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:18.916871 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:18.916874 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:18.917168 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:18.917388 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:18.917389 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:18.917390 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:18.917507 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:18.917783 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:18.917790 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:18.917964 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 19:09:18.918089 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 19:09:18.918261 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":102,"threads_started":1}
I20260925 19:09:18.919395 23223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.253" port: 35505 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 19:09:18 dist-test-slave-rn98 krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.252@KRBTEST.COM for kudu/127.0.196.253@KRBTEST.COM
Sep 25 19:09:18 dist-test-slave-rn98 krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:09:18.926687 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.919688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:35505 (local address 127.0.0.1:42708)
0925 19:09:18.919725 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:18.919735 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:18.919768 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:18.920031 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:18.920032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:18.920169 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:18.920236 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:18.920238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.920880 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.920881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:18.921376 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:18.921378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.921783 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.921785 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:18.921863 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:18.922263 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:18.922268 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:18.923676 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:18.925573 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:18.925574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:18.925574 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:18.925737 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:18.925963 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:18.925964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:18.925964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:18.926015 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:18.926305 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:18.926307 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:18.926356 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 19:09:18.926441 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 19:09:18.926539 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 19:09:18.926699 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.919848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42708 (local address 127.0.196.253:35505)
0925 19:09:18.919905 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:18.919907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:18.919915 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:09:18.919940 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:18.919941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:18.919956 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:18.920026 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:18.920328 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.920802 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.921468 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.921693 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.921779 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:18.923767 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:18.923769 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:18.923769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:18.923777 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:18.925493 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:18.925834 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:18.925835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:18.925835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:18.925898 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:18.926078 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:18.926079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:18.926079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:18.926227 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:18.926297 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:18.926484 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 19:09:18.926569 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 19:09:18.927204 23223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.252" port: 34119 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:09:18.927537 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.919559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48308 (local address 127.0.196.254:37985)
0925 19:09:18.919602 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:18.919604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:18.919640 (+ 36us) server_negotiation.cc:407] Connection header received
0925 19:09:18.919659 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:18.919660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:18.919672 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:18.919728 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:18.920332 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.920813 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.921624 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.921831 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.921918 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:18.924488 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:18.924490 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:18.924495 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:18.924504 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:18.926114 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:18.926479 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:18.926480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:18.926480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:18.926541 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:18.926972 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:18.926973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:18.926973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:18.927140 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:18.927313 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:18.927351 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 19:09:18.927430 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 19:09:18 dist-test-slave-rn98 krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.252@KRBTEST.COM for kudu/127.0.196.252@KRBTEST.COM
Sep 25 19:09:18 dist-test-slave-rn98 krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.253@KRBTEST.COM
I20260925 19:09:18.934407 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.927504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:34119 (local address 127.0.0.1:34542)
0925 19:09:18.927535 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:18.927545 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:18.927582 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:18.927921 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:18.927922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:18.928050 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:18.928078 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:18.928079 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.928702 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.928703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:18.929172 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:18.929173 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.929549 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.929550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:18.929598 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:18.930042 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:18.930047 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:18.931448 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:18.933173 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:18.933174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:18.933174 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:18.933390 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:18.933661 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:18.933661 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:18.933662 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:18.933801 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:18.934131 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:18.934132 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:18.934173 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 19:09:18.934245 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 19:09:18.934337 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260925 19:09:18.934530 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.927540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34542 (local address 127.0.196.252:34119)
0925 19:09:18.927816 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:18.927818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:18.927827 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:09:18.927847 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:18.927847 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:18.927858 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:18.927905 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:18.928156 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.928629 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.929251 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.929467 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.929551 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:18.931537 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:18.931539 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:18.931539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:18.931545 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:18.933079 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:18.933483 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:18.933484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:18.933485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:18.933549 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:18.933882 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:18.933883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:18.933884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:18.934033 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:18.934115 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:18.934291 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 19:09:18.934400 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":19}
I20260925 19:09:18.935683 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.928182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42718 (local address 127.0.196.253:35505)
0925 19:09:18.928217 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:18.928219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:18.928226 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:09:18.928286 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:18.928287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:18.928296 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:18.928338 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:18.928814 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.929274 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.929979 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.930190 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.930280 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:18.932566 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:18.932568 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:18.932568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:18.932575 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:18.934218 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:18.934931 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:18.934933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:18.934937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:18.935008 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:18.935231 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:18.935231 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:18.935231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:18.935336 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:18.935449 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:18.935558 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 19:09:18.935620 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:09:18.936735 23223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c65ab242345f45f69c5361db189b6f0a: Bootstrap starting.
I20260925 19:09:18.937587 787 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:09:18.937853 23223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c65ab242345f45f69c5361db189b6f0a: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:18.938200 23223 log.cc:824] T 00000000000000000000000000000000 P c65ab242345f45f69c5361db189b6f0a: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:18.939107 23223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c65ab242345f45f69c5361db189b6f0a: No bootstrap required, opened a new log
I20260925 19:09:18.938952 23045 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b8d9e0153112454c8dbc2e7e74333e13"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48308
I20260925 19:09:18.940706 23223 raft_consensus.cc:348] T 00000000000000000000000000000000 P c65ab242345f45f69c5361db189b6f0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d9e0153112454c8dbc2e7e74333e13" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37985 } } peers { permanent_uuid: "f51885b9308e4f1ca58ec53f93827dae" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 35505 } } peers { permanent_uuid: "c65ab242345f45f69c5361db189b6f0a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 34119 } }
I20260925 19:09:18.940848 23223 raft_consensus.cc:729] T 00000000000000000000000000000000 P c65ab242345f45f69c5361db189b6f0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c65ab242345f45f69c5361db189b6f0a, State: Initialized, Role: FOLLOWER
I20260925 19:09:18.940941 23223 consensus_queue.cc:260] T 00000000000000000000000000000000 P c65ab242345f45f69c5361db189b6f0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d9e0153112454c8dbc2e7e74333e13" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37985 } } peers { permanent_uuid: "f51885b9308e4f1ca58ec53f93827dae" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 35505 } } peers { permanent_uuid: "c65ab242345f45f69c5361db189b6f0a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 34119 } }
I20260925 19:09:18.941285 23223 sys_catalog.cc:564] T 00000000000000000000000000000000 P c65ab242345f45f69c5361db189b6f0a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:18.941283 23229 sys_catalog.cc:455] T 00000000000000000000000000000000 P c65ab242345f45f69c5361db189b6f0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d9e0153112454c8dbc2e7e74333e13" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37985 } } peers { permanent_uuid: "f51885b9308e4f1ca58ec53f93827dae" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 35505 } } peers { permanent_uuid: "c65ab242345f45f69c5361db189b6f0a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 34119 } } }
I20260925 19:09:18.941425 23229 sys_catalog.cc:458] T 00000000000000000000000000000000 P c65ab242345f45f69c5361db189b6f0a [sys.catalog]: This master's current role is: FOLLOWER
W20260925 19:09:18.943230 23240 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c65ab242345f45f69c5361db189b6f0a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:09:18.943290 23240 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:09:18.943594 23155 master_runner.cc:186] Error getting master registration for 127.0.196.254:37985: OK, instance_id { permanent_uuid: "b8d9e0153112454c8dbc2e7e74333e13" instance_seqno: 1790363358597338 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 19:09:18.945688 23155 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 19:09:18.945960 23073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.252" port: 34119 } attempt: 2
Sep 25 19:09:18 dist-test-slave-rn98 krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for kudu/127.0.196.252@KRBTEST.COM
I20260925 19:09:18.954702 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.946307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34556 (local address 127.0.196.252:34119)
0925 19:09:18.946354 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:18.946356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:18.946412 (+ 56us) server_negotiation.cc:407] Connection header received
0925 19:09:18.946435 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:18.946435 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:18.946443 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:18.946513 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:18.946876 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.947617 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.948648 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.948992 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.949073 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:18.951546 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:18.951547 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:18.951548 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:18.951568 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:18.953292 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:18.953763 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:18.953765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:18.953766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:18.953849 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:18.954064 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:18.954065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:18.954065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:18.954250 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:18.954336 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:18.954492 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 19:09:18.954578 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:09:18.954710 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.946239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:34119 (local address 127.0.0.1:34556)
0925 19:09:18.946330 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:18.946344 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:18.946384 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:18.946524 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:18.946525 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:18.946746 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:18.946800 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:18.946802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.947715 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.947716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:18.948538 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:18.948540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.949093 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.949094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:18.949154 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:18.949695 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:18.949701 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:18.951415 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:18.953421 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:18.953427 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:18.953428 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:18.953679 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:18.953933 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:18.953935 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:18.953935 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:18.954000 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:18.954330 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:18.954330 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:18.954381 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 19:09:18.954452 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 19:09:18.954557 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 19:09:18.956416 23073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8d9e0153112454c8dbc2e7e74333e13: Bootstrap starting.
I20260925 19:09:18.957062 23073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8d9e0153112454c8dbc2e7e74333e13: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:18.957370 23073 log.cc:824] T 00000000000000000000000000000000 P b8d9e0153112454c8dbc2e7e74333e13: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:18.958168 23073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8d9e0153112454c8dbc2e7e74333e13: No bootstrap required, opened a new log
I20260925 19:09:18.959270 23073 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8d9e0153112454c8dbc2e7e74333e13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d9e0153112454c8dbc2e7e74333e13" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37985 } } peers { permanent_uuid: "f51885b9308e4f1ca58ec53f93827dae" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 35505 } } peers { permanent_uuid: "c65ab242345f45f69c5361db189b6f0a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 34119 } }
I20260925 19:09:18.959421 23073 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8d9e0153112454c8dbc2e7e74333e13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8d9e0153112454c8dbc2e7e74333e13, State: Initialized, Role: FOLLOWER
I20260925 19:09:18.959534 23073 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8d9e0153112454c8dbc2e7e74333e13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d9e0153112454c8dbc2e7e74333e13" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37985 } } peers { permanent_uuid: "f51885b9308e4f1ca58ec53f93827dae" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 35505 } } peers { permanent_uuid: "c65ab242345f45f69c5361db189b6f0a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 34119 } }
I20260925 19:09:18.959852 23073 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8d9e0153112454c8dbc2e7e74333e13 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:18.959898 23045 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8d9e0153112454c8dbc2e7e74333e13 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 19:09:18.959977 23045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8d9e0153112454c8dbc2e7e74333e13 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:18.960800 23045 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8d9e0153112454c8dbc2e7e74333e13 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d9e0153112454c8dbc2e7e74333e13" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37985 } } peers { permanent_uuid: "f51885b9308e4f1ca58ec53f93827dae" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 35505 } } peers { permanent_uuid: "c65ab242345f45f69c5361db189b6f0a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 34119 } }
I20260925 19:09:18.961077 23045 leader_election.cc:288] T 00000000000000000000000000000000 P b8d9e0153112454c8dbc2e7e74333e13 [CANDIDATE]: Term 1 election: Requested vote from peers f51885b9308e4f1ca58ec53f93827dae (127.0.196.253:35505), c65ab242345f45f69c5361db189b6f0a (127.0.196.252:34119)
I20260925 19:09:18.961367 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8d9e0153112454c8dbc2e7e74333e13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d9e0153112454c8dbc2e7e74333e13" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37985 } } peers { permanent_uuid: "f51885b9308e4f1ca58ec53f93827dae" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 35505 } } peers { permanent_uuid: "c65ab242345f45f69c5361db189b6f0a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 34119 } } }
I20260925 19:09:18.961544 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8d9e0153112454c8dbc2e7e74333e13 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:09:18.961615 23197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8d9e0153112454c8dbc2e7e74333e13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c65ab242345f45f69c5361db189b6f0a"
I20260925 19:09:18.961774 23197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c65ab242345f45f69c5361db189b6f0a [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:18.963112 23197 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c65ab242345f45f69c5361db189b6f0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8d9e0153112454c8dbc2e7e74333e13 in term 1.
I20260925 19:09:18.963368 23021 leader_election.cc:302] T 00000000000000000000000000000000 P b8d9e0153112454c8dbc2e7e74333e13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8d9e0153112454c8dbc2e7e74333e13, c65ab242345f45f69c5361db189b6f0a; no voters:
I20260925 19:09:18.963501 23243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8d9e0153112454c8dbc2e7e74333e13 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:18.963675 23243 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8d9e0153112454c8dbc2e7e74333e13 [term 1 LEADER]: Becoming Leader. State: Replica: b8d9e0153112454c8dbc2e7e74333e13, State: Running, Role: LEADER
I20260925 19:09:18.963791 23243 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8d9e0153112454c8dbc2e7e74333e13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d9e0153112454c8dbc2e7e74333e13" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37985 } } peers { permanent_uuid: "f51885b9308e4f1ca58ec53f93827dae" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 35505 } } peers { permanent_uuid: "c65ab242345f45f69c5361db189b6f0a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 34119 } }
I20260925 19:09:18.964257 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8d9e0153112454c8dbc2e7e74333e13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8d9e0153112454c8dbc2e7e74333e13. Latest consensus state: current_term: 1 leader_uuid: "b8d9e0153112454c8dbc2e7e74333e13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d9e0153112454c8dbc2e7e74333e13" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37985 } } peers { permanent_uuid: "f51885b9308e4f1ca58ec53f93827dae" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 35505 } } peers { permanent_uuid: "c65ab242345f45f69c5361db189b6f0a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 34119 } } }
I20260925 19:09:18.964344 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8d9e0153112454c8dbc2e7e74333e13 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:18.964545 23264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:18.965057 23264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:18.963718 23005 master_runner.cc:186] Error getting master registration for 127.0.196.253:35505: OK, instance_id { permanent_uuid: "f51885b9308e4f1ca58ec53f93827dae" instance_seqno: 1790363358752219 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 19:09:18.966524 23005 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 19:09:18.968695 23197 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c65ab242345f45f69c5361db189b6f0a [term 1 FOLLOWER]: Refusing update from remote peer b8d9e0153112454c8dbc2e7e74333e13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:18.969873 23262 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b8d9e0153112454c8dbc2e7e74333e13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c65ab242345f45f69c5361db189b6f0a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:18.971797 23229 sys_catalog.cc:455] T 00000000000000000000000000000000 P c65ab242345f45f69c5361db189b6f0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8d9e0153112454c8dbc2e7e74333e13. Latest consensus state: current_term: 1 leader_uuid: "b8d9e0153112454c8dbc2e7e74333e13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d9e0153112454c8dbc2e7e74333e13" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37985 } } peers { permanent_uuid: "f51885b9308e4f1ca58ec53f93827dae" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 35505 } } peers { permanent_uuid: "c65ab242345f45f69c5361db189b6f0a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 34119 } } }
I20260925 19:09:18.971940 23229 sys_catalog.cc:458] T 00000000000000000000000000000000 P c65ab242345f45f69c5361db189b6f0a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:09:18.972105 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.962548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48320 (local address 127.0.196.254:37985)
0925 19:09:18.962588 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:18.962590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:18.963024 (+ 434us) server_negotiation.cc:407] Connection header received
0925 19:09:18.963126 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:18.963127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:18.963139 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:18.963172 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:18.964156 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.964882 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.965967 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.966240 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.966294 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:18.968073 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:18.968075 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:18.968075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:18.968083 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:18.970137 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:18.970606 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:18.970607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:18.970608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:18.970697 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:18.970903 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:18.970904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:18.970905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:18.971048 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:18.971139 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:18.971919 (+ 780us) server_negotiation.cc:299] Negotiation successful
0925 19:09:18.972013 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:09:18.977012 23262 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8d9e0153112454c8dbc2e7e74333e13 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8d9e0153112454c8dbc2e7e74333e13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d9e0153112454c8dbc2e7e74333e13" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37985 } } peers { permanent_uuid: "f51885b9308e4f1ca58ec53f93827dae" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 35505 } } peers { permanent_uuid: "c65ab242345f45f69c5361db189b6f0a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 34119 } } }
I20260925 19:09:18.977151 23262 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8d9e0153112454c8dbc2e7e74333e13 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:18.977779 23229 sys_catalog.cc:455] T 00000000000000000000000000000000 P c65ab242345f45f69c5361db189b6f0a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8d9e0153112454c8dbc2e7e74333e13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d9e0153112454c8dbc2e7e74333e13" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37985 } } peers { permanent_uuid: "f51885b9308e4f1ca58ec53f93827dae" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 35505 } } peers { permanent_uuid: "c65ab242345f45f69c5361db189b6f0a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 34119 } } }
I20260925 19:09:18.977902 23229 sys_catalog.cc:458] T 00000000000000000000000000000000 P c65ab242345f45f69c5361db189b6f0a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:09:18.977967 23266 mvcc.cc:205] Tried to move back new op lower bound from 7333328318332149760 to 7333328318316548096. Current Snapshot: MvccSnapshot[applied={T|T < 7333328318332149760}]
I20260925 19:09:18.978714 23264 catalog_manager.cc:1347] Generated new cluster ID: f0f44cec9bf74e9aa43bd10c10db014f
I20260925 19:09:18.978767 23264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:09:18.980706 23267 mvcc.cc:205] Tried to move back new op lower bound from 7333328318332149760 to 7333328318316548096. Current Snapshot: MvccSnapshot[applied={T|T < 7333328318332149760}]
I20260925 19:09:18.983582 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.962542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42720 (local address 127.0.196.253:35505)
0925 19:09:18.962588 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:18.962590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:18.969940 (+ 7350us) server_negotiation.cc:407] Connection header received
0925 19:09:18.970932 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:18.970934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:18.970948 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:18.970982 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:18.972788 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.973612 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.974829 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.975183 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.975247 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:18.978530 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:18.978532 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:18.978533 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:18.978542 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:18.980611 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:18.982288 (+ 1677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:18.982289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:18.982290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:18.982370 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:18.982734 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:18.982735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:18.982736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:18.982898 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:18.982990 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:18.983384 (+ 394us) server_negotiation.cc:299] Negotiation successful
0925 19:09:18.983482 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:09:18.993095 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.962841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34570 (local address 127.0.196.252:34119)
0925 19:09:18.962879 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:18.962881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:18.972683 (+ 9802us) server_negotiation.cc:407] Connection header received
0925 19:09:18.972845 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:18.972846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:18.972858 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:18.972932 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:18.974272 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.975059 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.986024 (+ 10965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.986345 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.986391 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:18.989128 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:18.989130 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:18.989130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:18.989140 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:18.991058 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:18.991820 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:18.991821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:18.991822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:18.991884 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:18.992240 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:18.992240 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:18.992241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:18.992388 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:18.992478 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:18.992892 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 19:09:18.992979 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:09:18.993268 23148 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.252" port: 34119 } attempt: 3
I20260925 19:09:18.997138 23264 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:18.998034 23264 catalog_manager.cc:1504] Loading token signing keys...
Sep 25 19:09:18 dist-test-slave-rn98 krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.253@KRBTEST.COM for kudu/127.0.196.252@KRBTEST.COM
I20260925 19:09:19.002103 23264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8d9e0153112454c8dbc2e7e74333e13: Generated new TSK 0
I20260925 19:09:19.002337 23264 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:09:19.003777 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.993560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:34119 (local address 127.0.0.1:34586)
0925 19:09:18.993659 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:18.993672 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:18.993715 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:18.994012 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:18.994013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:18.994216 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:18.994269 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:18.994272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.995319 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.995321 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:18.996142 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:18.996145 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.996756 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.996757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:18.996837 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:18.997425 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:18.997431 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:19.000072 (+ 2641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:19.002315 (+ 2243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:19.002316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:19.002317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:19.002562 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:19.002975 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:19.002976 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:19.002977 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:19.003037 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:19.003439 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:19.003440 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:19.003499 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 19:09:19.003558 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 19:09:19.003666 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 19:09:19.006368 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:18.993795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34586 (local address 127.0.196.252:34119)
0925 19:09:18.993871 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:18.993874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:18.993881 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:09:18.993902 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:18.993903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:18.993912 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:18.993963 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:18.994368 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.995222 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.996262 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:18.996606 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:18.996664 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:19.000182 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:19.000183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:19.000184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:19.000192 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:19.002129 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:19.002724 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:19.002725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:19.002726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:19.002854 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:19.003111 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:19.003112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:19.003112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:19.003309 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:19.005989 (+ 2680us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:19.006059 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 19:09:19.006196 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 19:09:19.008436 23148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f51885b9308e4f1ca58ec53f93827dae: Bootstrap starting.
I20260925 19:09:19.009277 23148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f51885b9308e4f1ca58ec53f93827dae: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:19.009706 23148 log.cc:824] T 00000000000000000000000000000000 P f51885b9308e4f1ca58ec53f93827dae: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:19.011471 23148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f51885b9308e4f1ca58ec53f93827dae: No bootstrap required, opened a new log
I20260925 19:09:19.014787 23148 raft_consensus.cc:348] T 00000000000000000000000000000000 P f51885b9308e4f1ca58ec53f93827dae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d9e0153112454c8dbc2e7e74333e13" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37985 } } peers { permanent_uuid: "f51885b9308e4f1ca58ec53f93827dae" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 35505 } } peers { permanent_uuid: "c65ab242345f45f69c5361db189b6f0a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 34119 } }
I20260925 19:09:19.015144 23148 raft_consensus.cc:729] T 00000000000000000000000000000000 P f51885b9308e4f1ca58ec53f93827dae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f51885b9308e4f1ca58ec53f93827dae, State: Initialized, Role: FOLLOWER
I20260925 19:09:19.015362 23148 consensus_queue.cc:260] T 00000000000000000000000000000000 P f51885b9308e4f1ca58ec53f93827dae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d9e0153112454c8dbc2e7e74333e13" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37985 } } peers { permanent_uuid: "f51885b9308e4f1ca58ec53f93827dae" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 35505 } } peers { permanent_uuid: "c65ab242345f45f69c5361db189b6f0a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 34119 } }
I20260925 19:09:19.016083 23121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f51885b9308e4f1ca58ec53f93827dae [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:19.016037 23122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8d9e0153112454c8dbc2e7e74333e13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f51885b9308e4f1ca58ec53f93827dae"
I20260925 19:09:19.017047 23121 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f51885b9308e4f1ca58ec53f93827dae [term 1 FOLLOWER]: Refusing update from remote peer b8d9e0153112454c8dbc2e7e74333e13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:19.015980 23148 sys_catalog.cc:564] T 00000000000000000000000000000000 P f51885b9308e4f1ca58ec53f93827dae [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:19.017947 23122 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f51885b9308e4f1ca58ec53f93827dae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8d9e0153112454c8dbc2e7e74333e13 in term 1.
I20260925 19:09:19.018374 23243 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b8d9e0153112454c8dbc2e7e74333e13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f51885b9308e4f1ca58ec53f93827dae" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 35505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 19:09:19.021631 23284 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f51885b9308e4f1ca58ec53f93827dae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:09:19.021698 23284 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:09:19.022167 23273 sys_catalog.cc:455] T 00000000000000000000000000000000 P f51885b9308e4f1ca58ec53f93827dae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8d9e0153112454c8dbc2e7e74333e13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d9e0153112454c8dbc2e7e74333e13" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37985 } } peers { permanent_uuid: "f51885b9308e4f1ca58ec53f93827dae" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 35505 } } peers { permanent_uuid: "c65ab242345f45f69c5361db189b6f0a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 34119 } } }
I20260925 19:09:19.022298 23273 sys_catalog.cc:458] T 00000000000000000000000000000000 P f51885b9308e4f1ca58ec53f93827dae [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:09:19.022367 23273 sys_catalog.cc:455] T 00000000000000000000000000000000 P f51885b9308e4f1ca58ec53f93827dae [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8d9e0153112454c8dbc2e7e74333e13. Latest consensus state: current_term: 1 leader_uuid: "b8d9e0153112454c8dbc2e7e74333e13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d9e0153112454c8dbc2e7e74333e13" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37985 } } peers { permanent_uuid: "f51885b9308e4f1ca58ec53f93827dae" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 35505 } } peers { permanent_uuid: "c65ab242345f45f69c5361db189b6f0a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 34119 } } }
I20260925 19:09:19.022442 23273 sys_catalog.cc:458] T 00000000000000000000000000000000 P f51885b9308e4f1ca58ec53f93827dae [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:09:19.022488 23273 sys_catalog.cc:455] T 00000000000000000000000000000000 P f51885b9308e4f1ca58ec53f93827dae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8d9e0153112454c8dbc2e7e74333e13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d9e0153112454c8dbc2e7e74333e13" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37985 } } peers { permanent_uuid: "f51885b9308e4f1ca58ec53f93827dae" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 35505 } } peers { permanent_uuid: "c65ab242345f45f69c5361db189b6f0a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 34119 } } }
I20260925 19:09:19.022559 23273 sys_catalog.cc:458] T 00000000000000000000000000000000 P f51885b9308e4f1ca58ec53f93827dae [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:09:19.027315 23286 mvcc.cc:205] Tried to move back new op lower bound from 7333328318467317760 to 7333328318316548096. Current Snapshot: MvccSnapshot[applied={T|T < 7333328318332149760}]
W20260925 19:09:19.040077 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:19.038537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42730 (local address 127.0.196.253:35505)
0925 19:09:19.038583 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:19.038586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:19.039277 (+ 691us) server_negotiation.cc:407] Connection header received
0925 19:09:19.039455 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:19.039457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:19.039469 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:19.039502 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:19.040003 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42730: BlockingRecv error: recv got EOF from 127.0.0.1:42730 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 19:09:19.041015 23077 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:19.038415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48334 (local address 127.0.196.254:37985)
0925 19:09:19.038468 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:19.038470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:19.040450 (+ 1980us) server_negotiation.cc:407] Connection header received
0925 19:09:19.040551 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:19.040552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:19.040563 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:19.040594 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:19.040928 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48334: BlockingRecv error: recv got EOF from 127.0.0.1:48334 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260925 19:09:19.041946 23227 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:19.038982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34592 (local address 127.0.196.252:34119)
0925 19:09:19.039041 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:19.039043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:19.041394 (+ 2351us) server_negotiation.cc:407] Connection header received
0925 19:09:19.041520 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:19.041521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:19.041531 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:19.041560 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:19.041888 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34592: BlockingRecv error: recv got EOF from 127.0.0.1:34592 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 19:09:19.044696 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:19.043946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42732 (local address 127.0.196.253:35505)
0925 19:09:19.043987 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:19.043989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:19.044135 (+ 146us) server_negotiation.cc:407] Connection header received
0925 19:09:19.044243 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:19.044244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:19.044264 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:19.044293 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:19.044633 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42732: BlockingRecv error: recv got EOF from 127.0.0.1:42732 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 19:09:19.048075 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:19.047213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42738 (local address 127.0.196.253:35505)
0925 19:09:19.047250 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:19.047251 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:19.047529 (+ 278us) server_negotiation.cc:407] Connection header received
0925 19:09:19.047634 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:19.047635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:19.047647 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:19.047678 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:19.048022 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42738: BlockingRecv error: recv got EOF from 127.0.0.1:42738 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 19:09:19.052176 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:19.051379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42754 (local address 127.0.196.253:35505)
0925 19:09:19.051415 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:19.051417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:19.051662 (+ 245us) server_negotiation.cc:407] Connection header received
0925 19:09:19.051757 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:19.051758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:19.051769 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:19.051797 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:19.052123 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42754: BlockingRecv error: recv got EOF from 127.0.0.1:42754 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 19:09:19.058557 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:19.057601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42770 (local address 127.0.196.253:35505)
0925 19:09:19.057642 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:19.057645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:19.057937 (+ 292us) server_negotiation.cc:407] Connection header received
0925 19:09:19.058047 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:19.058048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:19.058062 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:19.058094 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:19.058497 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42770: BlockingRecv error: recv got EOF from 127.0.0.1:42770 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 19:09:19.069120 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:19.068074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42782 (local address 127.0.196.253:35505)
0925 19:09:19.068120 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:19.068122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:19.068433 (+ 311us) server_negotiation.cc:407] Connection header received
0925 19:09:19.068546 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:19.068547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:19.068561 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:19.068593 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:19.069056 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42782: BlockingRecv error: recv got EOF from 127.0.0.1:42782 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 19:09:19.087680 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:19.086679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42790 (local address 127.0.196.253:35505)
0925 19:09:19.086727 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:19.086730 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:19.087043 (+ 313us) server_negotiation.cc:407] Connection header received
0925 19:09:19.087153 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:19.087154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:19.087166 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:19.087199 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:19.087614 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42790: BlockingRecv error: recv got EOF from 127.0.0.1:42790 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 19:09:19.122237 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:19.121093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42796 (local address 127.0.196.253:35505)
0925 19:09:19.121139 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:19.121141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:19.121610 (+ 469us) server_negotiation.cc:407] Connection header received
0925 19:09:19.121715 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:19.121716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:19.121729 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:19.121759 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:19.122173 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42796: BlockingRecv error: recv got EOF from 127.0.0.1:42796 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 19:09:19.188819 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:19.188050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42798 (local address 127.0.196.253:35505)
0925 19:09:19.188112 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:19.188115 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:19.188261 (+ 146us) server_negotiation.cc:407] Connection header received
0925 19:09:19.188300 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:19.188302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:19.188314 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:19.188400 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:19.188757 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42798: BlockingRecv error: recv got EOF from 127.0.0.1:42798 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260925 19:09:19.318984 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:19.318158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42812 (local address 127.0.196.253:35505)
0925 19:09:19.318251 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:19.318253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:19.318304 (+ 51us) server_negotiation.cc:407] Connection header received
0925 19:09:19.318366 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:19.318367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:19.318377 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:19.318448 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:19.318912 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42812: BlockingRecv error: recv got EOF from 127.0.0.1:42812 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 19:09:19.522974 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8d9e0153112454c8dbc2e7e74333e13 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8d9e0153112454c8dbc2e7e74333e13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8d9e0153112454c8dbc2e7e74333e13" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37985 } } peers { permanent_uuid: "f51885b9308e4f1ca58ec53f93827dae" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 35505 } } peers { permanent_uuid: "c65ab242345f45f69c5361db189b6f0a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 34119 } } }
I20260925 19:09:19.523154 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8d9e0153112454c8dbc2e7e74333e13 [sys.catalog]: This master's current role is: LEADER
W20260925 19:09:19.577049 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:19.576287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42828 (local address 127.0.196.253:35505)
0925 19:09:19.576369 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:19.576372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:19.576402 (+ 30us) server_negotiation.cc:407] Connection header received
0925 19:09:19.576467 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:19.576468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:19.576479 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:19.576529 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:19.576975 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42828: BlockingRecv error: recv got EOF from 127.0.0.1:42828 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 19:09:19.943753 23240 catalog_manager.cc:1259] Loaded cluster ID: f0f44cec9bf74e9aa43bd10c10db014f
I20260925 19:09:19.943861 23240 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c65ab242345f45f69c5361db189b6f0a: loading cluster ID for follower catalog manager: success
I20260925 19:09:19.944698 23240 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c65ab242345f45f69c5361db189b6f0a: acquiring CA information for follower catalog manager: success
I20260925 19:09:19.945043 23240 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c65ab242345f45f69c5361db189b6f0a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 19:09:20.022233 23284 catalog_manager.cc:1259] Loaded cluster ID: f0f44cec9bf74e9aa43bd10c10db014f
I20260925 19:09:20.022349 23284 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f51885b9308e4f1ca58ec53f93827dae: loading cluster ID for follower catalog manager: success
I20260925 19:09:20.023242 23284 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f51885b9308e4f1ca58ec53f93827dae: acquiring CA information for follower catalog manager: success
I20260925 19:09:20.023576 23284 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f51885b9308e4f1ca58ec53f93827dae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 19:09:20.093289 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:20.090412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42840 (local address 127.0.196.253:35505)
0925 19:09:20.090545 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:20.090547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:20.090558 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:09:20.090635 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:20.090635 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:20.090689 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:20.090774 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:20.090926 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:20.091509 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:20.092411 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:20.092613 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:20.092665 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:20.092848 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:20.092978 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:20.093062 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:20.093096 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 19:09:20.093121 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":82,"threads_started":1}
I20260925 19:09:20.098289 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:20.095565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34606 (local address 127.0.196.252:34119)
0925 19:09:20.095687 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:20.095689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:20.095696 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:09:20.095744 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:20.095745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:20.095802 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:20.095878 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:20.096039 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:20.096551 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:20.097337 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:20.097557 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:20.097630 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:20.097842 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:20.097961 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:20.098033 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:20.098113 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 19:09:20.098135 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":70,"threads_started":1}
I20260925 19:09:20.103221 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:20.100357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34616 (local address 127.0.196.252:34119)
0925 19:09:20.100399 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:20.100401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:20.100520 (+ 119us) server_negotiation.cc:407] Connection header received
0925 19:09:20.100592 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:20.100593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:20.100605 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:20.100655 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:20.100831 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:20.101338 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:20.102303 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:20.102692 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:20.102770 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:20.102943 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:20.102980 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:20.103034 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:20.103103 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 19:09:20.103129 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:09:20.103626 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:20.100226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42844 (local address 127.0.196.253:35505)
0925 19:09:20.100291 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:20.100294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:20.100449 (+ 155us) server_negotiation.cc:407] Connection header received
0925 19:09:20.100564 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:20.100565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:20.100580 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:20.100724 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:20.100906 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:20.101727 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:20.102708 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:20.103067 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:20.103128 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:20.103312 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:20.103360 (+ 48us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:20.103392 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:20.103522 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 19:09:20.103550 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 19:09:20.104048 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:20.100224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48338 (local address 127.0.196.254:37985)
0925 19:09:20.100492 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:20.100494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:20.100502 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:09:20.100557 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:20.100558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:20.100618 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:20.100711 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:20.100846 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:20.101776 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:20.102888 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:20.103219 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:20.103287 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:20.103515 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:20.103643 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:20.103715 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:20.103924 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 19:09:20.103955 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":76,"threads_started":1}
I20260925 19:09:20.106051 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 23005
I20260925 19:09:20.113353 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 23080
I20260925 19:09:20.118840 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 23155
2026-09-25T19:09:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2837 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:09:20 dist-test-slave-rn98 krb5kdc[23376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:09:20 dist-test-slave-rn98 krb5kdc[23376](info): set up 2 sockets
Sep 25 19:09:20 dist-test-slave-rn98 krb5kdc[23376](info): commencing operation
krb5kdc: starting...
W20260925 19:09:21.155673 787 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/krb5kdc/test-user.keytab.
Sep 25 19:09:21 dist-test-slave-rn98 krb5kdc[23376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:09:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:09:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.196.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
Entry for principal kudu/127.0.196.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab.
I20260925 19:09:21.277631 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:37979
--webserver_interface=127.0.196.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:41939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:37979,127.0.196.253:38467,127.0.196.252:40199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:09:21.359838 23389 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 19:09:21.360123 23389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:21.360162 23389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:21.360205 23389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:21.361413 23389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:21.361490 23389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:21.361513 23389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:21.361534 23389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:21.361560 23389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:21.363112 23389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:41939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:37979,127.0.196.253:38467,127.0.196.252:40199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/krb5kdc/kudu_127.0.196.254.keytab
--principal=kudu/127.0.196.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.254:37979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.196.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:21.363422 23389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:21.363665 23389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:21.368623 23394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:21.368643 23397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:21.368906 23389 server_base.cc:1034] running on GCE node
W20260925 19:09:21.368635 23395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:21.369222 23389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:21.369526 23389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:21.370738 23389 hybrid_clock.cc:648] HybridClock initialized: now 1790363361370732 us; error 39 us; skew 500 ppm
Sep 25 19:09:21 dist-test-slave-rn98 krb5kdc[23376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:21.373339 23389 init.cc:377] Logged in from keytab as kudu/127.0.196.254@KRBTEST.COM (short username kudu)
I20260925 19:09:21.373844 23389 webserver.cc:460] Webserver started at http://127.0.196.254:41117/ using document root <none> and password file <none>
I20260925 19:09:21.374076 23389 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:21.374153 23389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:21.374305 23389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:21.375465 23389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-0/data/instance:
uuid: "f0bdef9789db4fa4b43595e9d5b59ec7"
format_stamp: "Formatted at 2026-09-25 19:09:21 on dist-test-slave-rn98"
server_key: "0f3ea318f292ea5979377b80cdeff5c6"
server_key_iv: "9b68004e9889319c886f50c798f6e605"
server_key_version: "encryptionkey@0"
I20260925 19:09:21.375845 23389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-0/wal/instance:
uuid: "f0bdef9789db4fa4b43595e9d5b59ec7"
format_stamp: "Formatted at 2026-09-25 19:09:21 on dist-test-slave-rn98"
server_key: "0f3ea318f292ea5979377b80cdeff5c6"
server_key_iv: "9b68004e9889319c886f50c798f6e605"
server_key_version: "encryptionkey@0"
I20260925 19:09:21.377228 23389 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 19:09:21.378022 23404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:21.378218 23389 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:09:21.378327 23389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-0/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-0/wal
uuid: "f0bdef9789db4fa4b43595e9d5b59ec7"
format_stamp: "Formatted at 2026-09-25 19:09:21 on dist-test-slave-rn98"
server_key: "0f3ea318f292ea5979377b80cdeff5c6"
server_key_iv: "9b68004e9889319c886f50c798f6e605"
server_key_version: "encryptionkey@0"
I20260925 19:09:21.378405 23389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:21.400161 23389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:21.402689 23389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:21.402865 23389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:21.407112 23389 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.254:37979
I20260925 19:09:21.407150 23456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.254:37979 every 8 connection(s)
I20260925 19:09:21.407572 23389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-0/data/info.pb
I20260925 19:09:21.408592 23457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:21.410247 23457 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37979 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:09:21.412570 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 23389
I20260925 19:09:21.412694 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-0/wal/instance
I20260925 19:09:21.412940 787 external_mini_cluster.cc:1428] Setting key 25148932d8b8c073531d51aae7c5dfec
Sep 25 19:09:21 dist-test-slave-rn98 krb5kdc[23376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.196.253@KRBTEST.COM; defaulting to no policy
I20260925 19:09:21.422194 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.410977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:37979 (local address 127.0.0.1:48512)
0925 19:09:21.411234 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:21.411283 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:21.411869 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:21.412197 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:21.412200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:21.412491 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:21.412659 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:21.412664 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.414301 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.414303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:21.415524 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:21.415528 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.416157 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.416159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:21.416246 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:21.416874 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:21.416886 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:21.418553 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:21.420476 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:21.420482 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:21.420485 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:21.420765 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:21.421031 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:21.421032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:21.421033 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:21.421130 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:21.421538 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:21.421544 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:21.421612 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 19:09:21.421726 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 19:09:21.421979 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":110,"thread_start_us":118,"threads_started":1}
I20260925 19:09:21.422194 23460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.411056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48512 (local address 127.0.196.254:37979)
0925 19:09:21.411244 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:21.411254 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:21.411393 (+ 139us) server_negotiation.cc:407] Connection header received
0925 19:09:21.412045 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:21.412047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:21.412064 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:21.412186 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:21.412818 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.414163 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.415671 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.416020 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.416115 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:21.418682 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:21.418690 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:21.418690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:21.418698 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:21.420367 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.420869 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.420872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.420873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.420961 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.421198 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.421199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.421199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.421462 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:21.421560 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:21.421759 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 19:09:21.421930 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":136,"thread_start_us":68,"threads_started":1}
I20260925 19:09:21.423197 23457 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.253" port: 38467 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.253@KRBTEST.COM" created.
I20260925 19:09:21.423674 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.423525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:38467 (local address 127.0.0.1:47390)
0925 19:09:21.423593 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:21.423645 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.253:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 19:09:21.423856 23408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.196.253:38467: connect: Connection refused (error 111)
I20260925 19:09:21.424391 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.424262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:38467 (local address 127.0.0.1:47400)
0925 19:09:21.424321 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:21.424354 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.253:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 19:09:21.424597 23457 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.196.253:38467: Network error: Client connection negotiation failed: client connection to 127.0.196.253:38467: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/krb5kdc/kudu_127.0.196.253.keytab.
Entry for principal kudu/127.0.196.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/krb5kdc/kudu_127.0.196.253.keytab.
I20260925 19:09:21.440086 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.253:38467
--webserver_interface=127.0.196.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/krb5kdc/kudu_127.0.196.253.keytab
--principal=kudu/127.0.196.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:41939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:37979,127.0.196.253:38467,127.0.196.252:40199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:09:21.473860 23457 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.253" port: 38467 } attempt: 1
I20260925 19:09:21.474416 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.474265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:38467 (local address 127.0.0.1:47404)
0925 19:09:21.474341 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:21.474384 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.253:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 19:09:21.474846 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.474735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:38467 (local address 127.0.0.1:47416)
0925 19:09:21.474790 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:21.474825 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.253:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 19:09:21.474960 23457 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.196.253:38467: Network error: Client connection negotiation failed: client connection to 127.0.196.253:38467: connect: Connection refused (error 111)
W20260925 19:09:21.534220 23464 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 19:09:21.534482 23464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:21.534520 23464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:21.534562 23464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:21.535827 23464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:21.535903 23464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:21.535930 23464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:21.535961 23464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:21.535988 23464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:21.537549 23464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:41939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:37979,127.0.196.253:38467,127.0.196.252:40199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/krb5kdc/kudu_127.0.196.253.keytab
--principal=kudu/127.0.196.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.253:38467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.196.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:21.537863 23464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:21.538107 23464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:21.543233 23470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:21.543255 23457 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.253" port: 38467 } attempt: 2
W20260925 19:09:21.543229 23472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:21.543202 23469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:21.543651 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.543527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:38467 (local address 127.0.0.1:47420)
0925 19:09:21.543589 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:21.543615 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.253:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 19:09:21.543654 23464 server_base.cc:1034] running on GCE node
I20260925 19:09:21.543946 23464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:21.544065 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.543976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:38467 (local address 127.0.0.1:47422)
0925 19:09:21.544010 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:21.544043 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.253:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260925 19:09:21.544211 23457 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.196.253:38467: Network error: Client connection negotiation failed: client connection to 127.0.196.253:38467: connect: Connection refused (error 111)
I20260925 19:09:21.544354 23464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:21.545519 23464 hybrid_clock.cc:648] HybridClock initialized: now 1790363361545501 us; error 28 us; skew 500 ppm
Sep 25 19:09:21 dist-test-slave-rn98 krb5kdc[23376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:21.548439 23464 init.cc:377] Logged in from keytab as kudu/127.0.196.253@KRBTEST.COM (short username kudu)
I20260925 19:09:21.548944 23464 webserver.cc:460] Webserver started at http://127.0.196.253:36075/ using document root <none> and password file <none>
I20260925 19:09:21.549199 23464 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:21.549248 23464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:21.549387 23464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:21.550571 23464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-1/data/instance:
uuid: "004d4bc7c45e4c66a59f8d756debc088"
format_stamp: "Formatted at 2026-09-25 19:09:21 on dist-test-slave-rn98"
server_key: "78ffe57f7114d7b074ed5e10873feeaf"
server_key_iv: "5868c10ee5db78ee82522e0eca94e755"
server_key_version: "encryptionkey@0"
I20260925 19:09:21.551031 23464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-1/wal/instance:
uuid: "004d4bc7c45e4c66a59f8d756debc088"
format_stamp: "Formatted at 2026-09-25 19:09:21 on dist-test-slave-rn98"
server_key: "78ffe57f7114d7b074ed5e10873feeaf"
server_key_iv: "5868c10ee5db78ee82522e0eca94e755"
server_key_version: "encryptionkey@0"
I20260925 19:09:21.552460 23464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:09:21.553247 23479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:21.553444 23464 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:09:21.553545 23464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-1/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-1/wal
uuid: "004d4bc7c45e4c66a59f8d756debc088"
format_stamp: "Formatted at 2026-09-25 19:09:21 on dist-test-slave-rn98"
server_key: "78ffe57f7114d7b074ed5e10873feeaf"
server_key_iv: "5868c10ee5db78ee82522e0eca94e755"
server_key_version: "encryptionkey@0"
I20260925 19:09:21.553642 23464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:21.574317 23464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:21.576596 23464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:21.576747 23464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:21.581151 23464 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.253:38467
I20260925 19:09:21.581283 23531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.253:38467 every 8 connection(s)
I20260925 19:09:21.581662 23464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-1/data/info.pb
I20260925 19:09:21.582835 23532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:21.584408 23532 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37979 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:09:21.586369 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 23464
I20260925 19:09:21.586469 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-1/wal/instance
I20260925 19:09:21.586689 787 external_mini_cluster.cc:1428] Setting key 52d5cf555b3efd9a5ec7743aad15c485
Sep 25 19:09:21 dist-test-slave-rn98 krb5kdc[23376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.253@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.196.252@KRBTEST.COM; defaulting to no policy
I20260925 19:09:21.595227 23460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.585224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48514 (local address 127.0.196.254:37979)
0925 19:09:21.585303 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:21.585305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:21.585528 (+ 223us) server_negotiation.cc:407] Connection header received
0925 19:09:21.585999 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:21.586000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:21.586014 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:21.586079 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:21.587175 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.587692 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.588797 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.589018 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.589109 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:21.591564 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:21.591566 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:21.591566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:21.591595 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:21.593396 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.594101 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.594102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.594102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.594158 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.594471 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.594472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.594472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.594589 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:21.594755 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:21.595035 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 19:09:21.595126 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 19:09:21.595434 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.585180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:37979 (local address 127.0.0.1:48514)
0925 19:09:21.585379 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:21.585425 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:21.585908 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:21.586300 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:21.586306 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:21.586748 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:21.587063 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:21.587071 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.587794 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.587795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:21.588695 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:21.588698 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.589139 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.589140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:21.589272 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:21.589919 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:21.589930 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:21.591455 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:21.593522 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:21.593565 (+ 43us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:21.593568 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:21.594008 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:21.594234 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:21.594234 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:21.594235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:21.594404 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:21.594723 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:21.594732 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:21.594893 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 19:09:21.595015 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 19:09:21.595203 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":111,"threads_started":1}
I20260925 19:09:21.596159 23532 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.253" port: 38467 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.196.252@KRBTEST.COM" created.
Sep 25 19:09:21 dist-test-slave-rn98 krb5kdc[23376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.253@KRBTEST.COM for kudu/127.0.196.253@KRBTEST.COM
I20260925 19:09:21.604298 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.596437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:38467 (local address 127.0.0.1:47436)
0925 19:09:21.596496 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:21.596507 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:21.596552 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:21.596846 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:21.596847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:21.596993 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:21.597040 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:21.597042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.598123 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.598124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:21.598774 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:21.598776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.599405 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.599406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:21.599494 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:21.599926 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:21.599933 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:21.601199 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:21.603154 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:21.603155 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:21.603155 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:21.603335 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:21.603559 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:21.603560 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:21.603560 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:21.603615 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:21.603917 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:21.603917 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:21.603984 (+ 67us) client_negotiation.cc:769] Sending connection context
0925 19:09:21.604053 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 19:09:21.604168 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 19:09:21.604380 23536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.596503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47436 (local address 127.0.196.253:38467)
0925 19:09:21.596682 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:21.596688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:21.596710 (+ 22us) server_negotiation.cc:407] Connection header received
0925 19:09:21.596735 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:21.596737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:21.596757 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:21.596846 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:21.597123 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.597991 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.598854 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.599323 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.599382 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:21.601288 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:21.601299 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:21.601299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:21.601305 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:21.603051 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.603429 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.603433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.603433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.603491 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.603683 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.603685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.603685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.603845 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:21.603940 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:21.604093 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 19:09:21.604247 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":134,"thread_start_us":109,"threads_started":1}
I20260925 19:09:21.604990 23532 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.252" port: 40199 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:09:21.605365 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.605239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:40199 (local address 127.0.0.1:33270)
0925 19:09:21.605289 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:21.605335 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:40199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 19:09:21.605505 23481 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.196.252:40199: connect: Connection refused (error 111)
I20260925 19:09:21.605908 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.605810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:40199 (local address 127.0.0.1:33272)
0925 19:09:21.605863 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:21.605886 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:40199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 19:09:21.606015 23532 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.196.252:40199: Network error: Client connection negotiation failed: client connection to 127.0.196.252:40199: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.196.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/krb5kdc/kudu_127.0.196.252.keytab.
Entry for principal kudu/127.0.196.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/krb5kdc/kudu_127.0.196.252.keytab.
I20260925 19:09:21.615129 787 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhGKzmd/build/release/bin/kudu
/tmp/dist-test-taskhGKzmd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.252:40199
--webserver_interface=127.0.196.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/krb5kdc/kudu_127.0.196.252.keytab
--principal=kudu/127.0.196.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.196.212:41939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.196.254:37979,127.0.196.253:38467,127.0.196.252:40199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:09:21.635376 23457 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.253" port: 38467 } attempt: 3
Sep 25 19:09:21 dist-test-slave-rn98 krb5kdc[23376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for kudu/127.0.196.253@KRBTEST.COM
I20260925 19:09:21.643333 23536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.635961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47440 (local address 127.0.196.253:38467)
0925 19:09:21.636040 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:21.636042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:21.636080 (+ 38us) server_negotiation.cc:407] Connection header received
0925 19:09:21.636113 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:21.636114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:21.636124 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:21.636243 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:21.636633 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.637123 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.637868 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.638096 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.638188 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:21.640412 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:21.640414 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:21.640414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:21.640423 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:21.641988 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.642395 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.642397 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.642397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.642465 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.642734 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.642735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.642736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.642868 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:21.642955 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:21.643137 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 19:09:21.643232 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 19:09:21.643333 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.635855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:38467 (local address 127.0.0.1:47440)
0925 19:09:21.635949 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:21.635959 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:21.636036 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:21.636254 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:21.636255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:21.636440 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:21.636513 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:21.636514 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.637226 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.637228 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:21.637772 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:21.637774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.638178 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.638179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:21.638265 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:21.638873 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:21.638878 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:21.640309 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:21.642099 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:21.642101 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:21.642102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:21.642308 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:21.642544 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:21.642545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:21.642545 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:21.642658 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:21.642944 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:21.642945 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:21.643003 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 19:09:21.643093 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 19:09:21.643218 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 19:09:21.643911 23457 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.252" port: 40199 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:09:21.644336 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.644215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:40199 (local address 127.0.0.1:33276)
0925 19:09:21.644275 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:21.644309 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:40199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 19:09:21.644786 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.644657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:40199 (local address 127.0.0.1:33278)
0925 19:09:21.644740 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:21.644767 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:40199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260925 19:09:21.644904 23457 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.196.252:40199: Network error: Client connection negotiation failed: client connection to 127.0.196.252:40199: connect: Connection refused (error 111)
I20260925 19:09:21.655246 23532 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.252" port: 40199 } attempt: 1
I20260925 19:09:21.655752 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.655605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:40199 (local address 127.0.0.1:33292)
0925 19:09:21.655671 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:21.655718 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:40199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 19:09:21.656251 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.656119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:40199 (local address 127.0.0.1:33300)
0925 19:09:21.656176 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:21.656216 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:40199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 19:09:21.656411 23532 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.196.252:40199: Network error: Client connection negotiation failed: client connection to 127.0.196.252:40199: connect: Connection refused (error 111)
I20260925 19:09:21.676065 23457 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.252" port: 40199 } attempt: 1
I20260925 19:09:21.676656 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.676494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:40199 (local address 127.0.0.1:33314)
0925 19:09:21.676576 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:21.676622 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:40199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 19:09:21.677109 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.676976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:40199 (local address 127.0.0.1:33330)
0925 19:09:21.677048 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:21.677082 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:40199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 19:09:21.677266 23457 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.196.252:40199: Network error: Client connection negotiation failed: client connection to 127.0.196.252:40199: connect: Connection refused (error 111)
W20260925 19:09:21.701392 23539 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 19:09:21.701638 23539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:09:21.701666 23539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:09:21.701687 23539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:09:21.702966 23539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:09:21.703019 23539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:09:21.703034 23539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:09:21.703048 23539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:09:21.703061 23539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:09:21.704689 23539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.196.212:41939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.196.254:37979,127.0.196.253:38467,127.0.196.252:40199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/krb5kdc/kudu_127.0.196.252.keytab
--principal=kudu/127.0.196.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.196.252:40199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.196.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:09:21.704979 23539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:09:21.705240 23539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:09:21.710450 23544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:09:21.710461 23547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:21.710484 23539 server_base.cc:1034] running on GCE node
W20260925 19:09:21.710461 23545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:09:21.711107 23539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:09:21.711434 23539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:09:21.712632 23539 hybrid_clock.cc:648] HybridClock initialized: now 1790363361712627 us; error 82 us; skew 500 ppm
Sep 25 19:09:21 dist-test-slave-rn98 krb5kdc[23376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:09:21.715384 23539 init.cc:377] Logged in from keytab as kudu/127.0.196.252@KRBTEST.COM (short username kudu)
I20260925 19:09:21.715895 23539 webserver.cc:460] Webserver started at http://127.0.196.252:45765/ using document root <none> and password file <none>
I20260925 19:09:21.716097 23539 fs_manager.cc:359] Metadata directory not provided
I20260925 19:09:21.716140 23539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:09:21.716238 23539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:09:21.717344 23539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-2/data/instance:
uuid: "73292ec4a8c14802b9a64228302d8d6a"
format_stamp: "Formatted at 2026-09-25 19:09:21 on dist-test-slave-rn98"
server_key: "d49bb35e58a1ba18f137020443c974e5"
server_key_iv: "48e4dfe76c5ac04e9b41f519a646e00d"
server_key_version: "encryptionkey@0"
I20260925 19:09:21.717669 23539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-2/wal/instance:
uuid: "73292ec4a8c14802b9a64228302d8d6a"
format_stamp: "Formatted at 2026-09-25 19:09:21 on dist-test-slave-rn98"
server_key: "d49bb35e58a1ba18f137020443c974e5"
server_key_iv: "48e4dfe76c5ac04e9b41f519a646e00d"
server_key_version: "encryptionkey@0"
I20260925 19:09:21.719053 23539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:09:21.719738 23554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:09:21.719954 23539 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:09:21.720050 23539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-2/data,/tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-2/wal
uuid: "73292ec4a8c14802b9a64228302d8d6a"
format_stamp: "Formatted at 2026-09-25 19:09:21 on dist-test-slave-rn98"
server_key: "d49bb35e58a1ba18f137020443c974e5"
server_key_iv: "48e4dfe76c5ac04e9b41f519a646e00d"
server_key_version: "encryptionkey@0"
I20260925 19:09:21.720110 23539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:09:21.724596 23532 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.252" port: 40199 } attempt: 2
I20260925 19:09:21.725100 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.724948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:40199 (local address 127.0.0.1:33342)
0925 19:09:21.725019 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:21.725064 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:40199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 19:09:21.725489 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.725400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:40199 (local address 127.0.0.1:33346)
0925 19:09:21.725446 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:21.725472 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.252:40199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 19:09:21.725612 23532 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.196.252:40199: Network error: Client connection negotiation failed: client connection to 127.0.196.252:40199: connect: Connection refused (error 111)
I20260925 19:09:21.729710 23539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:09:21.731995 23539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:09:21.732172 23539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:09:21.736343 23539 rpc_server.cc:307] RPC server started. Bound to: 127.0.196.252:40199
I20260925 19:09:21.736411 23606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.196.252:40199 every 8 connection(s)
I20260925 19:09:21.736801 23539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-2/data/info.pb
I20260925 19:09:21.737907 23607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:09:21.739688 23607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37979 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:09:21.740123 787 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhGKzmd/build/release/bin/kudu as pid 23539
I20260925 19:09:21.740253 787 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhGKzmd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790363087086945-787-0/minicluster-data/master-2/wal/instance
I20260925 19:09:21.740482 787 external_mini_cluster.cc:1428] Setting key feb19974728b9032db1d282e69e35ecf
Sep 25 19:09:21 dist-test-slave-rn98 krb5kdc[23376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.252@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
Sep 25 19:09:21 dist-test-slave-rn98 krb5kdc[23376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.252@KRBTEST.COM
I20260925 19:09:21.750562 23460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.740514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48520 (local address 127.0.196.254:37979)
0925 19:09:21.740570 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:21.740572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:21.740974 (+ 402us) server_negotiation.cc:407] Connection header received
0925 19:09:21.741691 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:21.741692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:21.741706 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:21.741798 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:21.742338 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.742920 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.744011 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.744319 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.744400 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:21.747240 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:21.747241 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:21.747242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:21.747268 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:21.749002 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.749505 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.749507 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.749507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.749583 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.749850 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.749851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.749852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.749988 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:21.750070 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:21.750376 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 19:09:21.750466 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 19:09:21.750800 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.740526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.254:37979 (local address 127.0.0.1:48520)
0925 19:09:21.740804 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:21.740861 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:21.741526 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:21.741830 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:21.741832 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:21.742077 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:21.742240 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:21.742244 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.742997 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.742998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:21.743929 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:21.743931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.744451 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.744452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:21.744582 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:21.745314 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:21.745323 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:21.747136 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:21.749091 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:21.749098 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:21.749100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:21.749426 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:21.749656 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:21.749656 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:21.749657 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:21.749767 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:21.750062 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:21.750068 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:21.750212 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 19:09:21.750327 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 19:09:21.750524 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":22,"thread_start_us":102,"threads_started":1}
I20260925 19:09:21.751248 23611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.740886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33356 (local address 127.0.196.252:40199)
0925 19:09:21.741190 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:21.741195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:21.741243 (+ 48us) server_negotiation.cc:407] Connection header received
0925 19:09:21.741528 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:21.741537 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:21.741566 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:21.741657 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:21.742416 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.743498 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.744436 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.744893 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.744959 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:21.747815 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:21.747824 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:21.747825 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:21.747833 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:21.749638 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.749978 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.749981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.749981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.750041 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.750258 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.750259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.750259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.750429 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:21.750591 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:21.751053 (+ 462us) server_negotiation.cc:299] Negotiation successful
0925 19:09:21.751134 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":275,"thread_start_us":96,"threads_started":1}
I20260925 19:09:21.751372 23607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.253" port: 38467 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:09:21.752413 23457 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.252" port: 40199 } attempt: 2
Sep 25 19:09:21 dist-test-slave-rn98 krb5kdc[23376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.252@KRBTEST.COM for kudu/127.0.196.253@KRBTEST.COM
Sep 25 19:09:21 dist-test-slave-rn98 krb5kdc[23376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.254@KRBTEST.COM for kudu/127.0.196.252@KRBTEST.COM
Sep 25 19:09:21 dist-test-slave-rn98 krb5kdc[23376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.254@KRBTEST.COM
I20260925 19:09:21.759348 23536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.751755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47444 (local address 127.0.196.253:38467)
0925 19:09:21.751833 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:21.751835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:21.751843 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:09:21.751892 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:21.751893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:21.751905 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:21.751971 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:21.752230 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.752985 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.753925 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.754149 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.754242 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:21.756430 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:21.756432 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:21.756432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:21.756443 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:21.758090 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.758534 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.758535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.758536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.758610 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.758826 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.758827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.758827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.758952 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:21.759032 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:21.759193 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 19:09:21.759275 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 19:09:21.759400 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.751642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.253:38467 (local address 127.0.0.1:47444)
0925 19:09:21.751675 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:21.751687 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:21.751727 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:21.751973 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:21.751974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:21.752085 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:21.752153 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:21.752155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.753046 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.753047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:21.753828 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:21.753830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.754241 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.754242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:21.754308 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:21.754811 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:21.754816 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:21.756340 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:21.758188 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:21.758189 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:21.758190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:21.758457 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:21.758709 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:21.758709 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:21.758710 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:21.758774 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:21.759023 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:21.759024 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:21.759074 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 19:09:21.759157 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 19:09:21.759287 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260925 19:09:21.759788 23607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.196.252" port: 40199 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:09:21.761348 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.752665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:40199 (local address 127.0.0.1:33364)
0925 19:09:21.752708 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:21.752716 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:21.752751 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:21.752894 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:21.752895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:21.753045 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:21.753073 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:21.753074 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.753901 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.753902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:21.754728 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:21.754731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.755146 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.755147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:21.755219 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:21.755566 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:21.755571 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:21.757500 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:21.759950 (+ 2450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:21.759951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:21.759952 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:21.760438 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:21.760656 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:21.760657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:21.760658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:21.760729 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:21.761013 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:21.761014 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:21.761061 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 19:09:21.761151 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 19:09:21.761255 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 19:09:21.761343 23611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.752702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33364 (local address 127.0.196.252:40199)
0925 19:09:21.752748 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:21.752750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:21.752758 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:09:21.752834 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:21.752835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:21.752844 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:21.752893 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:21.753133 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.753816 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.754814 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.755065 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.755134 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:21.757615 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:21.757616 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:21.757616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:21.757623 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:21.759844 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.760507 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.760508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.760508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.760565 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.760806 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.760806 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.760807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.760933 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:21.761021 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:21.761178 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 19:09:21.761255 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 19:09:21.761857 23460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.752099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48528 (local address 127.0.196.254:37979)
0925 19:09:21.752167 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:21.752169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:21.752176 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:09:21.752196 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:21.752197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:21.752208 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:21.752263 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:21.752957 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.753499 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.754354 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.754603 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.754702 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:21.758159 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:21.758160 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:21.758161 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:21.758167 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:21.760422 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.760859 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.760860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.760861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.760904 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.761116 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.761118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.761118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.761253 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:21.761403 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:21.761658 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 19:09:21.761745 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 19:09:21.763404 23457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0bdef9789db4fa4b43595e9d5b59ec7: Bootstrap starting.
I20260925 19:09:21.764178 23457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0bdef9789db4fa4b43595e9d5b59ec7: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:21.764549 23457 log.cc:824] T 00000000000000000000000000000000 P f0bdef9789db4fa4b43595e9d5b59ec7: Log is configured to *not* fsync() on all Append() calls
Sep 25 19:09:21 dist-test-slave-rn98 krb5kdc[23376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.252@KRBTEST.COM for kudu/127.0.196.252@KRBTEST.COM
I20260925 19:09:21.766042 23457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0bdef9789db4fa4b43595e9d5b59ec7: No bootstrap required, opened a new log
Sep 25 19:09:21 dist-test-slave-rn98 krb5kdc[23376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.253@KRBTEST.COM
I20260925 19:09:21.768195 23457 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0bdef9789db4fa4b43595e9d5b59ec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bdef9789db4fa4b43595e9d5b59ec7" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37979 } } peers { permanent_uuid: "004d4bc7c45e4c66a59f8d756debc088" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 38467 } } peers { permanent_uuid: "73292ec4a8c14802b9a64228302d8d6a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 40199 } }
I20260925 19:09:21.768379 23457 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0bdef9789db4fa4b43595e9d5b59ec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0bdef9789db4fa4b43595e9d5b59ec7, State: Initialized, Role: FOLLOWER
I20260925 19:09:21.768481 23457 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0bdef9789db4fa4b43595e9d5b59ec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bdef9789db4fa4b43595e9d5b59ec7" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37979 } } peers { permanent_uuid: "004d4bc7c45e4c66a59f8d756debc088" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 38467 } } peers { permanent_uuid: "73292ec4a8c14802b9a64228302d8d6a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 40199 } }
I20260925 19:09:21.768798 23614 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0bdef9789db4fa4b43595e9d5b59ec7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bdef9789db4fa4b43595e9d5b59ec7" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37979 } } peers { permanent_uuid: "004d4bc7c45e4c66a59f8d756debc088" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 38467 } } peers { permanent_uuid: "73292ec4a8c14802b9a64228302d8d6a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 40199 } } }
I20260925 19:09:21.768883 23612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.760092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33378 (local address 127.0.196.252:40199)
0925 19:09:21.760240 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:21.760241 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:21.760246 (+ 5us) server_negotiation.cc:407] Connection header received
0925 19:09:21.760258 (+ 12us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:21.760258 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:21.760264 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:21.760339 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:21.760644 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.761824 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.762824 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.763679 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.763748 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:21.765815 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:21.765816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:21.765816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:21.765823 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:21.767598 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.767954 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.767955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.767955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.768008 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.768233 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.768234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.768234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.768424 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:21.768486 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:21.768657 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 19:09:21.768742 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":110,"threads_started":1}
I20260925 19:09:21.769152 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.760061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:40199 (local address 127.0.0.1:33378)
0925 19:09:21.760110 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:21.760119 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:21.760157 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:21.760341 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:21.760342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:21.760507 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:21.760548 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:21.760549 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.761912 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.761913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:21.762745 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:21.762748 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.763765 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.763766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:21.763818 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:21.764343 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:21.764348 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:21.765707 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:21.767699 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:21.767700 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:21.767700 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:21.767866 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:21.768111 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:21.768111 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:21.768112 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:21.768162 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:21.768507 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:21.768507 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:21.768547 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 19:09:21.768977 (+ 430us) client_negotiation.cc:240] Negotiation successful
0925 19:09:21.769071 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 19:09:21.768882 23457 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0bdef9789db4fa4b43595e9d5b59ec7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:21.771163 23607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73292ec4a8c14802b9a64228302d8d6a: Bootstrap starting.
I20260925 19:09:21.771770 23536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.762734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47458 (local address 127.0.196.253:38467)
0925 19:09:21.762810 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:21.762813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:21.762823 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:09:21.762867 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:21.762868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:21.762879 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:21.762944 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:21.763668 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.764380 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.765269 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.765569 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.765646 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:21.768542 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:21.768544 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:21.768544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:21.768553 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:21.770357 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.770815 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.770816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.770817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.770882 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.771083 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.771084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.771084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.771227 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:21.771382 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:21.771601 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 19:09:21.771684 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 19:09:21.771944 23607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73292ec4a8c14802b9a64228302d8d6a: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:21.770879 23614 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0bdef9789db4fa4b43595e9d5b59ec7 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:09:21.772257 23607 log.cc:824] T 00000000000000000000000000000000 P 73292ec4a8c14802b9a64228302d8d6a: Log is configured to *not* fsync() on all Append() calls
W20260925 19:09:21.771519 23625 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f0bdef9789db4fa4b43595e9d5b59ec7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:09:21.772317 23625 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:09:21.773099 23607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73292ec4a8c14802b9a64228302d8d6a: No bootstrap required, opened a new log
I20260925 19:09:21.772214 23389 master_runner.cc:186] Error getting master registration for 127.0.196.253:38467: OK, instance_id { permanent_uuid: "004d4bc7c45e4c66a59f8d756debc088" instance_seqno: 1790363361579712 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 19:09:21.775336 23607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73292ec4a8c14802b9a64228302d8d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bdef9789db4fa4b43595e9d5b59ec7" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37979 } } peers { permanent_uuid: "004d4bc7c45e4c66a59f8d756debc088" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 38467 } } peers { permanent_uuid: "73292ec4a8c14802b9a64228302d8d6a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 40199 } }
I20260925 19:09:21.775502 23389 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 19:09:21.775496 23607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73292ec4a8c14802b9a64228302d8d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73292ec4a8c14802b9a64228302d8d6a, State: Initialized, Role: FOLLOWER
I20260925 19:09:21.775590 23607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73292ec4a8c14802b9a64228302d8d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bdef9789db4fa4b43595e9d5b59ec7" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37979 } } peers { permanent_uuid: "004d4bc7c45e4c66a59f8d756debc088" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 38467 } } peers { permanent_uuid: "73292ec4a8c14802b9a64228302d8d6a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 40199 } }
I20260925 19:09:21.775697 787 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:09:21.775924 23607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73292ec4a8c14802b9a64228302d8d6a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:21.776623 23628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73292ec4a8c14802b9a64228302d8d6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bdef9789db4fa4b43595e9d5b59ec7" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37979 } } peers { permanent_uuid: "004d4bc7c45e4c66a59f8d756debc088" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 38467 } } peers { permanent_uuid: "73292ec4a8c14802b9a64228302d8d6a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 40199 } } }
I20260925 19:09:21.776679 23431 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f0bdef9789db4fa4b43595e9d5b59ec7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48528
I20260925 19:09:21.776757 23628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73292ec4a8c14802b9a64228302d8d6a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:09:21.776806 23431 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0bdef9789db4fa4b43595e9d5b59ec7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 19:09:21.776859 23431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0bdef9789db4fa4b43595e9d5b59ec7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:21.777895 23431 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0bdef9789db4fa4b43595e9d5b59ec7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bdef9789db4fa4b43595e9d5b59ec7" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37979 } } peers { permanent_uuid: "004d4bc7c45e4c66a59f8d756debc088" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 38467 } } peers { permanent_uuid: "73292ec4a8c14802b9a64228302d8d6a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 40199 } }
W20260925 19:09:21.777957 23639 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 73292ec4a8c14802b9a64228302d8d6a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:09:21.777995 23639 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:09:21.778151 23431 leader_election.cc:288] T 00000000000000000000000000000000 P f0bdef9789db4fa4b43595e9d5b59ec7 [CANDIDATE]: Term 1 election: Requested vote from peers 004d4bc7c45e4c66a59f8d756debc088 (127.0.196.253:38467), 73292ec4a8c14802b9a64228302d8d6a (127.0.196.252:40199)
I20260925 19:09:21.779160 23581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0bdef9789db4fa4b43595e9d5b59ec7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "73292ec4a8c14802b9a64228302d8d6a"
I20260925 19:09:21.779304 23581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73292ec4a8c14802b9a64228302d8d6a [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:21.780233 23581 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 73292ec4a8c14802b9a64228302d8d6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0bdef9789db4fa4b43595e9d5b59ec7 in term 1.
I20260925 19:09:21.780485 23406 leader_election.cc:302] T 00000000000000000000000000000000 P f0bdef9789db4fa4b43595e9d5b59ec7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73292ec4a8c14802b9a64228302d8d6a, f0bdef9789db4fa4b43595e9d5b59ec7; no voters:
I20260925 19:09:21.780643 23614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0bdef9789db4fa4b43595e9d5b59ec7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:09:21.780813 23614 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0bdef9789db4fa4b43595e9d5b59ec7 [term 1 LEADER]: Becoming Leader. State: Replica: f0bdef9789db4fa4b43595e9d5b59ec7, State: Running, Role: LEADER
I20260925 19:09:21.780907 23614 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0bdef9789db4fa4b43595e9d5b59ec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bdef9789db4fa4b43595e9d5b59ec7" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37979 } } peers { permanent_uuid: "004d4bc7c45e4c66a59f8d756debc088" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 38467 } } peers { permanent_uuid: "73292ec4a8c14802b9a64228302d8d6a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 40199 } }
I20260925 19:09:21.781467 23614 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0bdef9789db4fa4b43595e9d5b59ec7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0bdef9789db4fa4b43595e9d5b59ec7. Latest consensus state: current_term: 1 leader_uuid: "f0bdef9789db4fa4b43595e9d5b59ec7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bdef9789db4fa4b43595e9d5b59ec7" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37979 } } peers { permanent_uuid: "004d4bc7c45e4c66a59f8d756debc088" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 38467 } } peers { permanent_uuid: "73292ec4a8c14802b9a64228302d8d6a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 40199 } } }
I20260925 19:09:21.781600 23614 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0bdef9789db4fa4b43595e9d5b59ec7 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:21.782778 23649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:09:21.783151 23649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:09:21.783363 23539 master_runner.cc:186] Error getting master registration for 127.0.196.253:38467: OK, instance_id { permanent_uuid: "004d4bc7c45e4c66a59f8d756debc088" instance_seqno: 1790363361579712 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 19:09:21.786082 23539 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 19:09:21.788883 23460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.780376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48534 (local address 127.0.196.254:37979)
0925 19:09:21.780423 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:21.780425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:21.780435 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:09:21.780458 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:21.780459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:21.780468 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:21.780538 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:21.781331 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.782112 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.783322 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.783581 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.783626 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:21.785349 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:21.785351 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:21.785351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:21.785360 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:21.787171 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.787593 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.787594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.787595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.787650 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.788093 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.788094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.788094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.788276 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:21.788393 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:21.788537 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 19:09:21.788624 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:09:21.791721 23581 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 73292ec4a8c14802b9a64228302d8d6a [term 1 FOLLOWER]: Refusing update from remote peer f0bdef9789db4fa4b43595e9d5b59ec7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:21.792116 23647 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f0bdef9789db4fa4b43595e9d5b59ec7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73292ec4a8c14802b9a64228302d8d6a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 40199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:21.793798 23628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73292ec4a8c14802b9a64228302d8d6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0bdef9789db4fa4b43595e9d5b59ec7. Latest consensus state: current_term: 1 leader_uuid: "f0bdef9789db4fa4b43595e9d5b59ec7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bdef9789db4fa4b43595e9d5b59ec7" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37979 } } peers { permanent_uuid: "004d4bc7c45e4c66a59f8d756debc088" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 38467 } } peers { permanent_uuid: "73292ec4a8c14802b9a64228302d8d6a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 40199 } } }
I20260925 19:09:21.793915 23628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73292ec4a8c14802b9a64228302d8d6a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:09:21.795383 23536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.779502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47462 (local address 127.0.196.253:38467)
0925 19:09:21.779547 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:21.779549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:21.784392 (+ 4843us) server_negotiation.cc:407] Connection header received
0925 19:09:21.784496 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:21.784497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:21.784509 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:21.784540 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:21.785232 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.785967 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.787214 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.787522 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.787566 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:21.790465 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:21.790466 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:21.790467 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:21.790475 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:21.792508 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.794405 (+ 1897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.794407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.794407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.794473 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.794728 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.794729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.794730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.794874 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:21.794968 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:21.795181 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 19:09:21.795269 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:09:21.796814 23612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.784207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33386 (local address 127.0.196.252:40199)
0925 19:09:21.784251 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:21.784253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:21.788764 (+ 4511us) server_negotiation.cc:407] Connection header received
0925 19:09:21.788868 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:21.788869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:21.788881 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:21.788913 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:21.789598 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.790330 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.791365 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.791742 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.791791 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:21.793248 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:21.793249 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:21.793250 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:21.793257 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:21.794873 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.795575 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.795577 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.795577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.795638 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.796007 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.796008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.796009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.796161 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:21.796249 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:21.796635 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 19:09:21.796727 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:09:21.800479 23614 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0bdef9789db4fa4b43595e9d5b59ec7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f0bdef9789db4fa4b43595e9d5b59ec7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bdef9789db4fa4b43595e9d5b59ec7" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37979 } } peers { permanent_uuid: "004d4bc7c45e4c66a59f8d756debc088" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 38467 } } peers { permanent_uuid: "73292ec4a8c14802b9a64228302d8d6a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 40199 } } }
I20260925 19:09:21.800606 23614 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0bdef9789db4fa4b43595e9d5b59ec7 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:21.801025 23628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73292ec4a8c14802b9a64228302d8d6a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f0bdef9789db4fa4b43595e9d5b59ec7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bdef9789db4fa4b43595e9d5b59ec7" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37979 } } peers { permanent_uuid: "004d4bc7c45e4c66a59f8d756debc088" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 38467 } } peers { permanent_uuid: "73292ec4a8c14802b9a64228302d8d6a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 40199 } } }
I20260925 19:09:21.801124 23628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73292ec4a8c14802b9a64228302d8d6a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:09:21.801697 23649 catalog_manager.cc:1347] Generated new cluster ID: 3525bd67a0704023a79a6b472db595e8
I20260925 19:09:21.801753 23649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:09:21.801962 23650 mvcc.cc:205] Tried to move back new op lower bound from 7333328329895968768 to 7333328329855774720. Current Snapshot: MvccSnapshot[applied={T|T < 7333328329895968768 or (T in {7333328329895968768})}]
I20260925 19:09:21.802250 23652 mvcc.cc:205] Tried to move back new op lower bound from 7333328329895968768 to 7333328329855774720. Current Snapshot: MvccSnapshot[applied={T|T < 7333328329895968768}]
I20260925 19:09:21.816756 23532 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.252" port: 40199 } attempt: 3
Sep 25 19:09:21 dist-test-slave-rn98 krb5kdc[23376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.253@KRBTEST.COM for kudu/127.0.196.252@KRBTEST.COM
I20260925 19:09:21.822664 23649 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:09:21.823563 23649 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:09:21.826248 23612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.817156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33390 (local address 127.0.196.252:40199)
0925 19:09:21.817201 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:21.817203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:21.817214 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:09:21.817293 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:21.817294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:21.817304 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:21.817374 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:21.817737 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.818494 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.819594 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.819859 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.819962 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:21.822601 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:09:21.822602 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:09:21.822603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:09:21.822610 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:09:21.824962 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.825381 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.825382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.825382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.825467 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:09:21.825674 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:09:21.825675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:09:21.825675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:09:21.825818 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:09:21.825893 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:21.826060 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 19:09:21.826149 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:09:21.826263 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:21.817107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.196.252:40199 (local address 127.0.0.1:33390)
0925 19:09:21.817172 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 19:09:21.817184 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:09:21.817225 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:09:21.817396 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:09:21.817397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:09:21.817599 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:09:21.817658 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:21.817660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.818580 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.818581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:21.819479 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:09:21.819482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:09:21.819961 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:09:21.819962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:09:21.820027 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:21.820581 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:09:21.820587 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:09:21.822518 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:09:21.825043 (+ 2525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:21.825044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:21.825045 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:21.825299 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:21.825535 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:09:21.825535 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:09:21.825536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:09:21.825599 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:09:21.825893 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:09:21.825894 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:09:21.825953 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 19:09:21.826036 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 19:09:21.826158 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 19:09:21.828323 23532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 004d4bc7c45e4c66a59f8d756debc088: Bootstrap starting.
I20260925 19:09:21.829114 23532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 004d4bc7c45e4c66a59f8d756debc088: Neither blocks nor log segments found. Creating new log.
I20260925 19:09:21.829437 23532 log.cc:824] T 00000000000000000000000000000000 P 004d4bc7c45e4c66a59f8d756debc088: Log is configured to *not* fsync() on all Append() calls
I20260925 19:09:21.832907 23532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 004d4bc7c45e4c66a59f8d756debc088: No bootstrap required, opened a new log
I20260925 19:09:21.835713 23532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 004d4bc7c45e4c66a59f8d756debc088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bdef9789db4fa4b43595e9d5b59ec7" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37979 } } peers { permanent_uuid: "004d4bc7c45e4c66a59f8d756debc088" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 38467 } } peers { permanent_uuid: "73292ec4a8c14802b9a64228302d8d6a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 40199 } }
I20260925 19:09:21.835897 23532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 004d4bc7c45e4c66a59f8d756debc088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 004d4bc7c45e4c66a59f8d756debc088, State: Initialized, Role: FOLLOWER
I20260925 19:09:21.835986 23532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 004d4bc7c45e4c66a59f8d756debc088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bdef9789db4fa4b43595e9d5b59ec7" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37979 } } peers { permanent_uuid: "004d4bc7c45e4c66a59f8d756debc088" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 38467 } } peers { permanent_uuid: "73292ec4a8c14802b9a64228302d8d6a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 40199 } }
I20260925 19:09:21.836349 23532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 004d4bc7c45e4c66a59f8d756debc088 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:09:21.836472 23505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 004d4bc7c45e4c66a59f8d756debc088 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:09:21.837517 23505 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 004d4bc7c45e4c66a59f8d756debc088 [term 1 FOLLOWER]: Refusing update from remote peer f0bdef9789db4fa4b43595e9d5b59ec7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:09:21.837809 23649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0bdef9789db4fa4b43595e9d5b59ec7: Generated new TSK 0
I20260925 19:09:21.838059 23649 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:09:21.838001 23647 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f0bdef9789db4fa4b43595e9d5b59ec7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "004d4bc7c45e4c66a59f8d756debc088" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:09:21.838541 23657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 004d4bc7c45e4c66a59f8d756debc088 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bdef9789db4fa4b43595e9d5b59ec7" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37979 } } peers { permanent_uuid: "004d4bc7c45e4c66a59f8d756debc088" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 38467 } } peers { permanent_uuid: "73292ec4a8c14802b9a64228302d8d6a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 40199 } } }
I20260925 19:09:21.838660 23657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 004d4bc7c45e4c66a59f8d756debc088 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 19:09:21.839715 23668 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 004d4bc7c45e4c66a59f8d756debc088: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:09:21.839773 23668 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:09:21.837843 23506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0bdef9789db4fa4b43595e9d5b59ec7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "004d4bc7c45e4c66a59f8d756debc088"
I20260925 19:09:21.841503 23506 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 004d4bc7c45e4c66a59f8d756debc088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0bdef9789db4fa4b43595e9d5b59ec7 in term 1.
I20260925 19:09:21.848307 23657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 004d4bc7c45e4c66a59f8d756debc088 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0bdef9789db4fa4b43595e9d5b59ec7. Latest consensus state: current_term: 1 leader_uuid: "f0bdef9789db4fa4b43595e9d5b59ec7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bdef9789db4fa4b43595e9d5b59ec7" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37979 } } peers { permanent_uuid: "004d4bc7c45e4c66a59f8d756debc088" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 38467 } } peers { permanent_uuid: "73292ec4a8c14802b9a64228302d8d6a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 40199 } } }
I20260925 19:09:21.848408 23657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 004d4bc7c45e4c66a59f8d756debc088 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:09:21.848445 23657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 004d4bc7c45e4c66a59f8d756debc088 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f0bdef9789db4fa4b43595e9d5b59ec7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bdef9789db4fa4b43595e9d5b59ec7" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37979 } } peers { permanent_uuid: "004d4bc7c45e4c66a59f8d756debc088" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 38467 } } peers { permanent_uuid: "73292ec4a8c14802b9a64228302d8d6a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 40199 } } }
I20260925 19:09:21.848480 23657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 004d4bc7c45e4c66a59f8d756debc088 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:09:21.854178 23670 mvcc.cc:205] Tried to move back new op lower bound from 7333328330064711680 to 7333328329855774720. Current Snapshot: MvccSnapshot[applied={T|T < 7333328329895968768}]
W20260925 19:09:21.873950 23536 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:21.872436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47476 (local address 127.0.196.253:38467)
0925 19:09:21.872485 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:21.872488 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:21.873189 (+ 701us) server_negotiation.cc:407] Connection header received
0925 19:09:21.873302 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:21.873304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:21.873315 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:21.873348 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:21.873866 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47476: BlockingRecv error: recv got EOF from 127.0.0.1:47476 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260925 19:09:21.874809 23612 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:21.872894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33402 (local address 127.0.196.252:40199)
0925 19:09:21.872950 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:21.872952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:21.874214 (+ 1262us) server_negotiation.cc:407] Connection header received
0925 19:09:21.874318 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:21.874319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:21.874330 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:21.874361 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:21.874737 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33402: BlockingRecv error: recv got EOF from 127.0.0.1:33402 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 19:09:21.875864 23460 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:21.875158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48550 (local address 127.0.196.254:37979)
0925 19:09:21.875200 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:21.875202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:21.875326 (+ 124us) server_negotiation.cc:407] Connection header received
0925 19:09:21.875422 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:21.875423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:21.875432 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:21.875463 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:21.875797 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48550: BlockingRecv error: recv got EOF from 127.0.0.1:48550 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 19:09:21.879812 23536 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:21.878955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47478 (local address 127.0.196.253:38467)
0925 19:09:21.878997 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:21.879000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:21.879198 (+ 198us) server_negotiation.cc:407] Connection header received
0925 19:09:21.879317 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:21.879318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:21.879341 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:21.879371 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:21.879754 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47478: BlockingRecv error: recv got EOF from 127.0.0.1:47478 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 19:09:21.883316 23536 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:21.882324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47494 (local address 127.0.196.253:38467)
0925 19:09:21.882366 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:21.882368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:21.882699 (+ 331us) server_negotiation.cc:407] Connection header received
0925 19:09:21.882852 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:21.882853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:21.882867 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:21.882894 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:21.883256 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47494: BlockingRecv error: recv got EOF from 127.0.0.1:47494 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 19:09:21.887428 23536 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:21.886520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47510 (local address 127.0.196.253:38467)
0925 19:09:21.886559 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:21.886561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:21.886865 (+ 304us) server_negotiation.cc:407] Connection header received
0925 19:09:21.886969 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:21.886970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:21.886982 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:21.887013 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:21.887370 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47510: BlockingRecv error: recv got EOF from 127.0.0.1:47510 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 19:09:21.893601 23536 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:21.892684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47526 (local address 127.0.196.253:38467)
0925 19:09:21.892729 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:21.892731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:21.893016 (+ 285us) server_negotiation.cc:407] Connection header received
0925 19:09:21.893123 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:21.893124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:21.893137 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:21.893167 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:21.893540 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47526: BlockingRecv error: recv got EOF from 127.0.0.1:47526 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 19:09:21.904050 23536 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:21.903075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47536 (local address 127.0.196.253:38467)
0925 19:09:21.903123 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:21.903125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:21.903424 (+ 299us) server_negotiation.cc:407] Connection header received
0925 19:09:21.903532 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:21.903534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:21.903546 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:21.903577 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:21.903975 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47536: BlockingRecv error: recv got EOF from 127.0.0.1:47536 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 19:09:21.922468 23536 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:21.921462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47552 (local address 127.0.196.253:38467)
0925 19:09:21.921508 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:21.921510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:21.921832 (+ 322us) server_negotiation.cc:407] Connection header received
0925 19:09:21.921951 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:21.921952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:21.921965 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:21.921996 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:21.922408 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47552: BlockingRecv error: recv got EOF from 127.0.0.1:47552 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 19:09:21.957001 23536 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:21.955912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47556 (local address 127.0.196.253:38467)
0925 19:09:21.955959 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:21.955962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:21.956284 (+ 322us) server_negotiation.cc:407] Connection header received
0925 19:09:21.956393 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:21.956394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:21.956409 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:21.956440 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:21.956921 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47556: BlockingRecv error: recv got EOF from 127.0.0.1:47556 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 19:09:22.023379 23536 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:22.022553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47568 (local address 127.0.196.253:38467)
0925 19:09:22.022659 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:22.022662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:22.022788 (+ 126us) server_negotiation.cc:407] Connection header received
0925 19:09:22.022823 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:22.022825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:22.022838 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:22.022929 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:22.023299 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47568: BlockingRecv error: recv got EOF from 127.0.0.1:47568 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 19:09:22.153456 23536 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:22.152625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47584 (local address 127.0.196.253:38467)
0925 19:09:22.152737 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:22.152739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:22.152814 (+ 75us) server_negotiation.cc:407] Connection header received
0925 19:09:22.152918 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:22.152919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:22.152933 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:22.153005 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:22.153386 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47584: BlockingRecv error: recv got EOF from 127.0.0.1:47584 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 19:09:22.411408 23536 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:09:22.410680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47588 (local address 127.0.196.253:38467)
0925 19:09:22.410749 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:22.410751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:22.410764 (+ 13us) server_negotiation.cc:407] Connection header received
0925 19:09:22.410860 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:22.410861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:22.410879 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:22.410941 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:09:22.411344 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47588: BlockingRecv error: recv got EOF from 127.0.0.1:47588 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 19:09:22.457988 23647 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0bdef9789db4fa4b43595e9d5b59ec7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f0bdef9789db4fa4b43595e9d5b59ec7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bdef9789db4fa4b43595e9d5b59ec7" member_type: VOTER last_known_addr { host: "127.0.196.254" port: 37979 } } peers { permanent_uuid: "004d4bc7c45e4c66a59f8d756debc088" member_type: VOTER last_known_addr { host: "127.0.196.253" port: 38467 } } peers { permanent_uuid: "73292ec4a8c14802b9a64228302d8d6a" member_type: VOTER last_known_addr { host: "127.0.196.252" port: 40199 } } }
I20260925 19:09:22.458177 23647 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0bdef9789db4fa4b43595e9d5b59ec7 [sys.catalog]: This master's current role is: LEADER
I20260925 19:09:22.778435 23639 catalog_manager.cc:1259] Loaded cluster ID: 3525bd67a0704023a79a6b472db595e8
I20260925 19:09:22.778522 23639 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 73292ec4a8c14802b9a64228302d8d6a: loading cluster ID for follower catalog manager: success
I20260925 19:09:22.779489 23639 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 73292ec4a8c14802b9a64228302d8d6a: acquiring CA information for follower catalog manager: success
I20260925 19:09:22.779861 23639 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 73292ec4a8c14802b9a64228302d8d6a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 19:09:22.840236 23668 catalog_manager.cc:1259] Loaded cluster ID: 3525bd67a0704023a79a6b472db595e8
I20260925 19:09:22.840319 23668 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 004d4bc7c45e4c66a59f8d756debc088: loading cluster ID for follower catalog manager: success
I20260925 19:09:22.841326 23668 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 004d4bc7c45e4c66a59f8d756debc088: acquiring CA information for follower catalog manager: success
I20260925 19:09:22.841684 23668 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 004d4bc7c45e4c66a59f8d756debc088: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 19:09:22.927699 23741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:22.924726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47598 (local address 127.0.196.253:38467)
0925 19:09:22.924852 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:22.924854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:22.924864 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:09:22.924894 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:22.924895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:22.924962 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:22.925058 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:22.925233 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:22.925761 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:22.926769 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:22.926970 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:22.927036 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:22.927226 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:22.927356 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:22.927429 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:22.927534 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 19:09:22.927556 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":54,"threads_started":1}
I20260925 19:09:22.932163 23747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:22.929787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33416 (local address 127.0.196.252:40199)
0925 19:09:22.929925 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:22.929926 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:22.929935 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:09:22.929971 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:22.929972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:22.930009 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:22.930085 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:22.930198 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:22.930762 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:22.931479 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:22.931669 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:22.931717 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:22.931866 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:22.931970 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:22.932027 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:22.932069 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 19:09:22.932085 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260925 19:09:22.937103 23747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:22.934063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33424 (local address 127.0.196.252:40199)
0925 19:09:22.934112 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:22.934114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:22.934347 (+ 233us) server_negotiation.cc:407] Connection header received
0925 19:09:22.934459 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:22.934460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:22.934471 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:22.934522 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:22.934718 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:22.935209 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:22.936304 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:22.936559 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:22.936591 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:22.936760 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:22.936793 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:22.936841 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:22.936983 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 19:09:22.937005 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:09:22.937328 23741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:22.934194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47600 (local address 127.0.196.253:38467)
0925 19:09:22.934240 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:22.934242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:22.934334 (+ 92us) server_negotiation.cc:407] Connection header received
0925 19:09:22.934405 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:22.934407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:22.934418 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:22.934465 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:22.934654 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:22.935404 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:22.936449 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:22.936766 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:22.936826 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:22.937043 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:22.937076 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:22.937102 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:22.937206 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 19:09:22.937234 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:09:22.937579 23754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:09:22.934066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48554 (local address 127.0.196.254:37979)
0925 19:09:22.934191 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 19:09:22.934211 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 19:09:22.934387 (+ 176us) server_negotiation.cc:407] Connection header received
0925 19:09:22.934437 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:09:22.934438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:09:22.934489 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:09:22.934565 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:09:22.934775 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:22.935536 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:22.936381 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:09:22.936688 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:09:22.936749 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:09:22.937055 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:09:22.937178 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:09:22.937233 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 19:09:22.937472 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 19:09:22.937498 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":66,"threads_started":1}
I20260925 19:09:22.946260 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 23389
I20260925 19:09:22.951850 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 23464
I20260925 19:09:22.956955 787 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhGKzmd/build/release/bin/kudu with pid 23539
2026-09-25T19:09:22Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2836 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5674 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (275877 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST